Two years running Tor relays: a retrospective
I spent around two years experimenting with and running Tor bridges and relays, until life got too busy and I had to put a pause to my efforts. Now that I’ve got some time again, and I’m back at it, I thought I’d write up a bit of a retro on the last two years.
In this post I’ll cover why and how I got started, the evolution of my relays and processes, and the lessons I learnt along the way.
The Process
The Beginning
I’ve always been interested in computers, programming, and security, so when I first heard about Tor years ago, I was instantly interested! I’m not sure if it was the first time I ever heard about Tor, but I distinctly remember hearing about it on an early episode of Security Now, when I started listening to the podcast in 2009. It was interesting from a technical stand point, and of course, what young university student wouldn’t be interested in this so called ‘dark web’?
Over the years I dabbled in running a Tor bridge or two, but always shied away from running an actual relay, not to mention I didn’t have the money to pay for a server. A few years ago, after moving to the U.K., and getting a job that would let me afford small tech projects, I decided it was time to properly support The Tor Project. I donated for a while, but wanted to get more involved in the interesting tech side.
Of course, I should mention, if I didn’t believe in the project’s mission, I wouldn’t have wasted my time and money on this:
To advance human rights and freedoms by creating and deploying free and open source anonymity and privacy technologies, supporting their unrestricted availability and use, and furthering their scientific and popular understanding.
The First Few Relays
I started by setting up a Tor bridge on a Digital Ocean droplet that I was paying for anyway, but hardly used. I had 1TB of bandwidth to use each month, that was just sitting there. Learning more about Tor relays, how to set them up, and how to setup a bridge and keep it maintained was my first challenge. This didn’t prove particularly difficult, and I soon also set up a bridge on an unused Raspberry Pi at home. This turned out to be fairly simple, as I had unlimited bandwidth, even though bridges tend to not use that much. Lookout for a follow up post on how to setup a Tor bridge, and specifically, on a Raspberry Pi. In the meantime, there are many great guides on how to do this just a search away.
 |
|---|
| My first Tor bridge, three years of history |
Once I was comfortable with bridges, I decided to try my hand at running a non-exit node Tor relay. Non-exit is important here, as exit nodes are where Tor users traffic egress onto the normal internet, and they often have trouble with law enforcement, which I wanted to avoid. I did some research on good providers to use. I also additional resources such as: Webhosts for Tor Relays Who Are Not; OVH, Hetzner or Scaleway, which is a bit dated but still useful, and Looking for virtual servers for 10 TOR exit nodes which proved more useful.
I decided to start with Host World, and went for one of their cheaper VPS options:
- VPS option: VPS 3
- Specs: 4 cores, 4GB memory, 1GB connection, unmetered bandwidth
Setting up this relay was a fun learning experience. I watched it slowly get more and more use, eventually peaking with traffic of around 25MB/s. After two weeks, I decided to setup another one on Host World, also using their VPS 3 offering. I went with Ubuntu, but down the line I decided that using Debian was a better option. Three months later, I decided to run one more on Host World, this time I went with Debian and their VPS 1 offering (2 core/1GB). I’d learned that Tor is mostly single threaded, and wanted to test how it performed with only 2 cores, and less memory. Turns out it worked just fine, but couldn’t handle as many connections or as much traffic. I discuss this in more detail in Lessons Learnt.
At this point I’d refined my process of setting up and configuring new relays, knew how to choose VPSs, and had taken a many notes! I decided to write my first blog post here, to provide an easy reference for myself, and hopefully also help others in the process.
Several More Relays!
Turns out, it’s quite addictive to see relays you’ve setup actually being used, gaining higher consensus weights, and using more and more bandwidth! Over the next three months I setup eight more relays! 😱 I tried some low cost ones, from cloud.co.za (2 core/4GB/100M), and Ionos (Linux XS, 1 core/1GB/1G). These proved less successful, no matter how low I limited the bandwidth on the Ionos VPS, it seemed to lock up and become unresponsive, given it only cost me ~£1 a month I wasn’t too bothered. The cloud.co.za one was perfectly stable, but even after a few months it received little traffic, perhaps the slow 100M shared connection was the problem?
I then discovered pfCloud and had quite a lot of fun with them. I started out with one VPS to test out their service in the Netherlands, they were cheap, around £6 for a 2 core, 4GB VPS with a 5G connection and unmetered (fair use) traffic. I eventually spun up three more servers with them, an AMD Epyc KVM 1 (1 core/2GB/10G), and two AMD Ryzen KVM 2’s (2 core/5GB/10G). For the specs, these were much cheaper than most of the other options I’d seen! At best, I saw just over 30MB/s of traffic on each of them. Their support was fairly also good, even if they did have some downtime and a security breach during my time using them. They’re cheap, and generally quite good, but perhaps not the most stable.
Finally, I’d discovered Incognet early on in my search for good relay hosts, however they were always quite expensive for what they offered. Then I came across a special, and I got two AMD Epyc VPSs (2 core/4GB/5G) with a 40TB bandwidth limit (20TB was the norm for them at the time), for just £7 a month. I did have to commit to both for a year, and pay up front, but this was still an excellent price. I judged that 40TB wouldn’t be too restrictive, as I’d rarely seen any of my existing relays use more than this in a month…
Turns out, I was wrong! Perhaps it was the location, or the hardware, but these two new relays received a large amount of traffic nearly from the beginning. It took me a few months of tweaking to get the bandwidth limits and cap correct, otherwise these two would easily exhaust their bandwidth within the first half of the month.
The Whole Family
At this point, I was spending around £100 a month running Tor relays for the fun of it! 💸 I decided to take a break from creating new ones, and rather consolidate my learnings. Throughout this process I’d learnt many new things, and started contributing in the Tor Project forum to help others and improve my own knowledge. My notes and spreadsheet of servers had also grown over time. I’d planned to post about it all sooner, but then life got in the way.
I learnt about things like the Tor Weather Project, which is a great service. You enter the details of your bridges and relays, and will be notified by email if any of them are offline for a certain amount of time. I found this very useful, as early on I often had to tweak bandwidth limits. Before this, I used Uptime Robot, which proved more versatile. I also learnt about AROI and OrNetStats, I’m planning a blog post about that soon, but for now the website explains it in detail.
I should note, I only ever ran bridges and non-exit node relays (middle/guard). I’d like to run an exit node one day, once I’ve done more research.
In summary, I was running the following for around two years:
| Host | Type | Hardware | Cost | Cores | Memory | Connection | Bandwidth | Location |
|---|---|---|---|---|---|---|---|---|
| Digital Ocean | Bridge | Basic Regular Intel | £6 | 1 | 1GB | 1G | 1TB | NL |
| Home | Bridge | Raspberry Pi 400 | Free | 4 | 4GB | 1G | Unmetered | UK |
| Host World | Relay | VPS 1 | £6 | 2 | 1GB | 1G | Unmetered | UK |
| Host World | Relay | VPS 3 | £14 | 4 | 4GB | 1G | Unmetered | UK |
| Host World | Relay | VPS 3 | £14 | 4 | 4GB | 1G | Unmetered | UK |
| Cloud.co.za | Relay | Cloud Server LN1 | £6 | 2 | 4GB | 100M | Unmetered | ZA |
| Ionos | Relay | Linux XS | £1 | 1 | 1GB | 1G | Unmetered | UK |
| pfCloud | Relay | Xeon KVM 2 | £6 | 2 | 4GB | 5G | Unmetered | NL |
| pfCloud | Relay | AMD Ryzen KVM 2 | £12 | 2 | 5GB | 10G | Unmetered | NL |
| pfCloud | Relay | AMD Ryzen KVM 2 | £12 | 2 | 5GB | 10G | Unmetered | NL |
| pfCloud | Relay | AMD Epyc KVM 1 | £6.50 | 1 | 2GB | 10G | Unmetered | NL |
| Incognet | Relay | AMD Epyc | £7 | 2 | 4GB | 5G | 40TB | US |
| Incognet | Relay | AMD Epyc | £7 | 2 | 4GB | 5G | 40TB | US |
Lessons Learnt
1. Processor, Memory, and Bandwidth
tl;dr: Order of priority when deciding on the specs of a relay: Unmetered bandwidth, single core performance, system memory (2GB min), then connection speed.
The most important lesson I learnt during this whole experience was the importance of single core CPU performance and the amount of memory when running a Tor relay. I went into it thinking that unmetered bandwidth and as fast a connection as possible were the most important considerations, but I was wrong.
I started out not setting any bandwidth cap or speed limits, and this worked fine, for a while… As a new relay goes through the usual life cycle, it gradually gets given more traffic. After many failed liveness checks, I found curious log entries, stating that my relay was overloaded. After some research, My relay or bridge is overloaded what does this mean? , I determined that either the CPU or memory were the bottleneck, and that I needed to limit the overall bandwidth of the relay. I referred to How can I limit the total amount of bandwidth used by my Tor relay? and What bandwidth shaping options are available to Tor relays?. At this point, I hadn’t seen any of my relays hit over 25MB/s.
I set the limits to 25MB/s, and kept an eye on things. The overloaded state meant my relay lost consensus weight, since it was unstable, so I had to wait a while for it to get more traffic again. I still had issues, and slowly lowered it, a few MB/s at a time. My initial relay seemed to stabilise at 20MB/s, while my second, lowered powered one, had to be reduced to around 12MB/s.
Further investigation of the system stats led me to find that when the load average was over 1, things started to become problematic. Memory was a lesser concern, but still important. 1GB didn’t seem like enough for anything over 12MB/s. This is likely due to the fact that a higher combined transfer rate generally translates to more connections through my relays, and the connection state table in memory would grow.
From then on, whenever I setup a new relay I’d always limit the speed to 20MB/s. After a few weeks, once it had stabilised and gained some traffic, I either lowered it if I got the dreaded overloaded message, or increased it slowly. Some relays I setup couldn’t even cope with 10MB/s, while others seemed to handle 35MB/s with room to spare.
I also discovered how the Tor service used by the relay was mostly single threaded, Threads in Tor, Guidance on optimal Tor relay server configurations:
“With 2 IPv4 addreses per relay as a hard limit, the biggest bottleneck you will encounter is that most of Tor’s code-base is singe-threaded, except for maybe onionskin decryption and compression of files.”
Which means that a single, fast, CPU core is much more useful than 4 underpowered cores.
For the rest of my relays, I generally prioritised single core CPU performance, opting to try the Ryzen VPSs offered by pfCloud. These worked wonderfully, and I sustained over 30MB/s, sometimes even seeing it go as high as 35MB/s. I also aimed for a minimum of 2GB of memory.
This is similar to what the official relay requirements document says, with a few differences:
A non-exit relay faster than 40 Mbit/s should have at least 1 GB of RAM.
I found this to be too little, or at least, since Linux also needs RAM, running a Tor relay and expecting over 20MB/s on a system with only 1GB of memory isn’t going to work.
Any modern CPU should be fine.
This is true, but only in a limited sense. For a slow relay, any modern CPU is probably fine. However, if you want your relay to run at a reasonable speed (>10MB/s), you need something with a little more single core power.
2. DDoS attacks
tl;dr: If you’re unlucky your relay may be DDoS’d to the point where it becomes unusable. Research Tor relay hardening tips, and apply those you find useful.
I’d heard about Tor relays being attacked in the past, but didn’t pay it much heed when I started setting up my relays. Then one day, one of my new pfCloud relays kept being reported as down. It didn’t have the usual overloaded symptoms, and but had a high load average. I found it had a very large number of connections, but hardly any traffic. None of my other relays were experiencing this issue, and it persisted even after a reboot, and leaving the server off for over a day. When it started back up, it become bogged down with connections once more.
Time for more research! I found reports from others who had similar issues in the past, and a DDoS attack on the relay seemed like the most likely culprit. I found some guides, tor-ddos and tangentially A beginner’s guide to Tor relay hardening, on hardening your Tor relay against these sorts of attacks. Some of these guides were more geared towards exit relays, but were helpful all the same. I made a few tweaks here and there in an attempt to remedy the situation. One solution I didn’t end up trying was to only allow traffic through the relay from other relays, and to exit nodes. This solution seemed overly complicated and was prone to becoming out of date if not maintained regularly.
Eventually, after a few days, my relay worked again and I didn’t spend too much more time thinking about it. Vowing to properly implement the solution described earlier, when I had the time.
3. Config Management and Backup
tl;dr: Always make a backup of your config files and relay keys, and develop a config management solution if you plan to run more than a few relays.
As the number of relays I was running grew, things were becoming a little complicated and difficult to manage. With 10+ systems, all with slightly different torrc files, and sometimes specific system tweaks, trying something new or knowing which relay was running what config, was fairly tedious. I also made use of the MyFamily config option, as any good operator should, and for this I had to keep track of the key fingerprints for all my relays, and update all config files each time I added a new one. Sometime soon this should be made easier by Happy Families.
I’d also had to migrate a relay or two over to new hosts, for example, when I realised the VPS I had chosen was underpowered, or saw a really good deal elsewhere. I could have just created new relays, but then I’d lose all of the reputation and consensus weight that my relay had earned. Turns out, this topic is covered in the Tor project’s official docs, I want to upgrade/move my relay. How do I keep the same identity?
I did more research, and found various projects on Tor configuration management, and notably, one which used Ansible to setup and configure Tor relays, Deploying Tor Relays within Minutes, Ansible Relayor. I didn’t end up using this solution, however it is a pretty good one if you’re willing to learn how to use it.
My solution was simple, I created an archive, tgz, of the main tor directories containing the config files and identity keys. I’d then scp this over to my own system, and store these in my usual backups. Your relays cryptographic identity keys are very important and should always be kept safe, as per the post install docs:
Only do this if you have a very secure place for your keys as if stolen, these keys could theoretically allow traffic decryption or impersonation.
A slightly better solution for configuration may have been to store the torrc files in a Git repository, that way I could have a history of changes I made to the files, and see why I had changed them.
Conclusion
In conclusion, I ran 10+ Tor relays, including bridges and entry/middle relay nodes, for around 2 years, and learnt some interesting lessons along the way.
I learnt a lot, and had even more fun, setting up these relays, keeping an eye on them, and tweaking them. And of course, looking out for good deals on new VPS’s 💸
I plan to get back into running relays soon, and hopefully learn even more. I might even get a dedicated server this time, something I’d always considered, but not pursued due to the high costs. With drastically more processing power and memory, just how much traffic could my relays support? ⚡️
If you’re considering it, take up the challenge and help support The Tor Project!