Advanced User Discussion: Caching Proxy Server

  • 1
  • Idea
  • Updated 2 years ago
For those of you out there who have some networking and/or Linux experience. I thought it would be worth creating a thread that talks about caching proxy servers to help conserve bandwidth and improve speeds.

Just a disclaimer, I'm assuming that anyone who attempts anything in this discussion is doing so at THEIR OWN RISK and assumes full responsibility in making any changes to their router and/or computers.

My first step was to expand the DNS cache on my router which is running DD-WRT. This should help improve page loads for frequently visited pages since we don't have to suffer the 600ms or so delay for the DNS lookup on frequently visited sites. For reference, I enabled DNSMasq in the Services tab under Setup, and added the line 'cache-size=2000' to Additional DNSMasq Options.

My second step was to setup a caching proxy server. I had a spare Raspberry Pi laying around and I decided to install Squid (http://www.squid-cache.org/) which seems to be the best open source / free option around for a caching proxy server. The downside is that it's quite complicated to get going.

Unfortunately, Raspbian (which is based off of Debian Wheezy) has a very old version of Squid in its repositories and so I ended up compiling the latest version of Squid from source by using a modified version of this tutorial (http://www.tonmann.com/2015/04/compile-squid-3-5-x-under-debian-jessie/) which is for a newer version of Debian. If anyone decides to attempt this, let me know and I'll post details of what I did.

You can run Squid on Windows and I think there are some tutorials out there. 

If anyone wants a copy of my Squid configuration file, let me know and I can post it as well.

Overall, after getting it all setup, it's worked quite well and I do notice the difference. I haven't looked at gathering any metrics yet since that's a bit involved with Squid but if I do, I'll definitely post the results here.

Otherwise, if anyone has setup any other caching proxy servers. It would be great to hear how things went.

Cheers,
Joshua
Photo of Joshua

Joshua

  • 210 Posts
  • 65 Reply Likes
  • happy

Posted 3 years ago

  • 1
Photo of J&J

J&J

  • 1687 Posts
  • 1003 Reply Likes
This is all fine but....   You cannot escape the ViaSat DNS, it will intercept and commandeer all DNS activity, even hand-typed IP addresses.  There's no way around it.  If you find a way post it, but their system is rather confined.
Photo of Joshua

Joshua

  • 210 Posts
  • 65 Reply Likes
Interesting.. If what you're saying is true, that's a pretty anti net neutrality practice because they would have to be spoofing IP addresses and modifying packets that contain the DNS responses.

Right now I'm just caching the results that I'm getting back from Exede's DNS servers. I haven't tried using Google's DNS servers to see if I get back different responses for IP addresses, etc...
Photo of Joshua

Joshua

  • 210 Posts
  • 65 Reply Likes
Just an update that I was able to verify the hijacking of DNS by ViaSat.

That just seems so... wrong.
(Edited)
Photo of J&J

J&J

  • 1687 Posts
  • 1003 Reply Likes
It's not a net neutrality thing.  ViaSat has to spool everything from your request to websites and pass it on to you as a stream.  If every site you went to had to wait for the TCP/IP overhead to do it's thing your speed would be about zero.  The time for RTS, CTS, IP headers, check-sums and the like would be dreadfully time consuming if it happened one packet at a time to and from your computer to the sites thus ViaSat spooling seemingly fakes out the sites and from our point of view the connection could have been next door instead of nearly 100,000 miles round trip.  Our latency is not so much to the sites but to the ViaSat ground station spool. ViaSat hijacks the DNS to insure all requested data arrives at their spool so we don't miss out on the speed enhancement realized by their system known as accelnet.  If you could find a way to bypass this system, you wouldn't want to use it.
(Edited)
Photo of Joshua

Joshua

  • 210 Posts
  • 65 Reply Likes
So that is why they have a disclaimer about poor VPN connectivity because they can't do any of the accellnet magic which causes the connection to be much slower.
Photo of J&J

J&J

  • 1687 Posts
  • 1003 Reply Likes
Yup, that about sums it up.
Photo of slowBill

slowBill

  • 54 Posts
  • 21 Reply Likes
Hello Joshua,
I would be interested in seeing your config file. 
is squid for Pi a single binary that you might share someplace for lazy users like my self.
I get frustrated and overwhelmed if  "apt-get install" does not get me where I want to be.
are you using the on-board SD for the cache or an external storage device? 
did you notice which change made the most improvement, DNS caching with DDwrt or squid proxy server?
Photo of Joshua

Joshua

  • 210 Posts
  • 65 Reply Likes
I can definitely get you my config file. Unfortunately, it isn't a single binary, I ended up compiling from source myself. There were a few tricks to it because Raspbian is based on an old version of Debian (wheezy).

I'll see what I can do to package it up into a .deb file that you can just install. Otherwise, I'll just zip the compile source directory and you should just be able to do a 'make install'.

I have a 32 GB SD card that I'm using for the cache and I haven't had any issues running out of space.

I'm actually doing DNS caching on both because squid has built in DNS caching too. But I think it's worth while doing DNS caching on your router if possible because that will hit all requests.
Photo of Joshua

Joshua

  • 210 Posts
  • 65 Reply Likes
Here is my squid.conf with all the comments removed. It might be a bit before I can see if I can generate a deb file for the latest version of squid. Just out of curiosity, are you running 3.1 on a Pi now? The binary that I compiled would only be valid for a Raspberry Pi or equivalent armv7l kernel. Hope this helps :)
acl localnet src 192.168.11.0/24
acl localnet src 192.168.1.0/24
acl SSL_ports port 443 #https
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT

http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access deny to_localhost
http_access allow localnet
http_access allow localhost
http_access deny all

http_port 3128

maximum_object_size_in_memory 512 KB
cache_dir ufs /var/spool/squid3 24000 16 256

maximum_object_size 96 MB
logfile_rotate 10
coredump_dir /var/spool/squid3
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
(Edited)
Photo of Joshua

Joshua

  • 210 Posts
  • 65 Reply Likes
Just an update that I've added the follow lines "ipcache_size 0" and "fqdncache_size 0" to disable DNS caching. I've also shut off DNS caching on DD-WRT.

I'm noticing issues with cached DNS lookups causing either slow page loads or unable to connect error messages. I believe this is related to Exede's hijacking of DNS. At the moment, every time I do a ping to Google, I get a different IP address.

I've made a separate post about this issue as well.
Photo of C0RR0SIVE

C0RR0SIVE

  • 41 Posts
  • 14 Reply Likes
The downfall of SQUID is that everyone is going to HTTPS, so for SQUID to remain functional beyond the few HTTP websites that exist, you will have to use a MiM and use your own local certificates...  I have been playing around with it for years on PFSense, and it's a REAL pain to get MiM to work when visiting the likes of Facebook or Google.

As for the whole speed thing... Yeah, not buying it, I know the competitor caches DNS at a modem level, but, you can specify your DNS server on your router or computer, only after the initial lookup does the modem do an intercept from the way things appear.  Never saw a speed decrease at all using a non-isp DNS server.
(Edited)
Photo of Joshua

Joshua

  • 210 Posts
  • 65 Reply Likes
You can test the DNS hijacking by utilizing a VPN and polling the same DNS server for a result along with testing multiple DNS servers without VPN. You should notice that, when not on a VPN, not only do your response times for the DNS lookups not vary but the results are the same too. Since your results are always the same, you would never see a change in performance because you're not actually using a different DNS server, it's just always Exede's results.

Yeah, I do agree that a caching proxy server is limited now by the number of sites utilizing HTTPS. However, tech news sites I do frequent on a daily basis are still unencrypted and contain a large amount of media.
Photo of Alijah Jean

Alijah Jean

  • 2 Posts
  • 0 Reply Likes
I want to use proxy server for my PS4 what do I do?
Photo of Alijah Jean

Alijah Jean

  • 2 Posts
  • 0 Reply Likes
I want to use proxy server for my PS4 what do I do?
Photo of Joshua

Joshua

  • 210 Posts
  • 65 Reply Likes
Hi Alijah,

Before getting down that path, what are you hoping to cache for your PS4?

Most caching proxy servers only work with standard non-encrypted HTTP traffic which means that any other kind of traffic (HTTPS, other TCP / UDP traffic) will basically just pass through a proxy server without doing anything.

Thanks,
Joshua
Photo of Matt F

Matt F, Viasat Official Rep

  • 114 Posts
  • 30 Reply Likes
New topic

Please reference the new conversation here: How to use PS4 as proxy server

This conversation is no longer open for comments or replies.