If you missed part 1, go here.
When I started building the server, I was (and still am) a fan of torrents. Using sites like ezrss.it, I could then automate the downloading of torrents with a simple RSS downloader script. Then I'd set up watch folders in rTorrent and it would take care of downloading and notifying me when complete.
That is, until I was told about Usenet.
Usenet, for the uninformed, is one of the oldest systems available on the Internet. Also know as user groups or discussion groups, Usenet servers store (up to a certain limit, called retention) messages posted by users who connect to them using specific software. You can get a feel for Usenet by using Google Groups, although technically Google's implementation is a little different and doesn't store binary files.
Now, due to post size limitations, these binary files need to be split into smaller chunks. And to prevent a lot of grief, each collection of posts containing chunks of a bigger file also contain a PAR file, which holds parity information to ensure that when all chunks are stitched together no bit is missing. It's like a photo ID for the file.
You can download and stitch the files together, then check the parity of the file against the PAR file, but that's a laborious process that can take a long chunk of your time for each download, so the best way here is to automate this process.
Enter SABnzbd+. The name may not make much sense, but this is the BEST way to automate downloads. There are plenty of docs on how to set it up on all different machines, but for Ubuntu all you need is follow this guide.
I'll give you some time to get your head around SABnzbd+. Go on. I'll wait.
Glueing It All Together, aka. The Solution
OK, so now that you know your way around SABnzbd+, all there is left is to download some NZB files and add them to the queue, right? Wrong!
As I said before, I'm a geek, so let's try and automate that as well. And before you think it's not worth it, imagine that I now can take a long holiday and come back to a home full of new TV episodes on my Apple TV, all converted and tagged for my viewing pleasure!
So with the hardware, the OS and the download manager in place, all we need is a way to automate the discovery and download of new NZB files. For that, we turn to an alpha piece of software called SickBeard.
Git, or How To Install SickBeard
The best way to install SickBeard is to get it straight from the developers. This, of course, can be a nuisance sometimes since not everything works out-of-the-box. For most of the time, however, it should work fine. If it doesn't, be sure to let the creators of SB know.
Once you decide where you want to install SickBeard's files (I have them in /opt/Sick-Beard), use the
git command to download the source files:
git clone http://github.com/midgetspy/Sick-Beard.git
This works similar to
svn checkout, and will create a folder with all the files needed inside. (If you don't know anything about svn, where have you been for the past 5 years? In a cave?)
From there, you can launch SB by running
python SickBeard.py. It's possible that it will try to launch your default browser, but just ignore it (you can disable that later in SB's config).
It may complain about port 8080 already being in use (by SABnzbd+, no less), so we need to edit a file called
config.ini inside SB's folder and change the
web_port setting to something else (I chose 8081). When done, save the file and try to launch SickBeard again.
Now open your browser on another machine and check that you can connect to the NAS' IP address on port 8081. If you can, keep on reading! Otherwise, try checking SickBeard's greatly detailed log window for clues as to what is missing.
There When You Need It
Just like SABnzbd+ is set to auto-start with the system, we want SickBeard to be available as well, since it will take care of the scraping of NZBs for new TV show episodes. Since we got SB from source, we'll need to use our own script to start/restart/stop it.
Luckily for you, it's been done. Save the script from the previous link as
/etc/init.d/sickbeard and assign execute permissions to it (
chmod 755 /etc/init.d/sickbeard).
Then edit the file on line 35 and replace the word
CHANGEME with your username. Also, be sure to read the comments on the file to understand how the stop operation works.
To aid with the above setup, a small script also needs to be created inside
/usr/local/bin. Just copy the contents from this link, taking care to change the path from
/opt/Sick-Beard to where you downloaded SB's files. Change the permissions of this script to 755 and run sudo update-rc.d sickbeard defaults to add SB to system startup.
The next step is maintaining it, and this will vary greatly from person to person. Go into SickBeard and add a few shows to it. SickBeard will create the folder for the TV show inside the path you specify, and save some metadata information in it as well.
With some TV shows added to SickBeard, make sure to test its ability to download the NZB file and pass it to SABnzbd+, which is configurable from SickBeard's Config > Episode Downloads page. You'll be asked for an API key, which you can find inside SABnzbd+'s Config > General page.
Once they are talking to each other, you can keep an eye on SABnzbd+'s queue to see the new downloads appear.
Enjoy your completely automated system (save from the ‘adding your own shows' part)!
Wait a Minute! What About Converting the Files to Play on the Apple TV?
Yes, I know I neglected to mention that, but only to entice your curiosity. In my scenario, I have to convert every single file from their MKV or AVI containers to an M4V iTunes-friendly format. I also have to convert DTS tracks to AC3, since the Apple TV can't understand DTS at all. I know, a PITA.
This process can take quite sometime to describe and solve (probably even warranting its own post), so I'll just leave some pointers here.
First, make sure to read up on SABnzbd+'s post-processing script functionality. I have created over the course of 1 year a script (still work-in-progress) available here that makes use of SAB's ability to run scripts after a download is finished. It basically works like this:
- SABnzbd+ finishes a download and calls the process_tv_episode.sh script from its configured user-scripts folder passing the folder where the file is as a parameter.
- The process_tv_episode.sh script checks for the presence of a DTS track and calls mkvdts2ac3.sh if it exists. If it doesn't, it jumps straight to step 4.
- The mkvdts2ac3.sh script converts the DTS track to an AC3 one and re-muxes the file.
- The process_tv_episode.sh script then attempts to connect to the MacBook Pro on my network to convert the file using HandBrake's CLI program. If my laptop is not available, it will take the job of converting the file on the NAS itself, albeit at a much slower speed (Core 2 Duo vs. Atom 330).
- Once the conversion is done, the process_tv_episode.sh script calls rename-file.py, which renames the file and tags it appropriately.
- After the files are renamed, they are moved to the appropriate Automatically Add to iTunes folder.
- SABnzbd+ acknowledges the finished process and sends me an email with a log of what's been done and when it finished. Neat!
If you're feeling adventurous, follow the links and see what you can come up with. My scripts are a good starting point.
Credits, or Who's to Blame
Couldn't have done it with the help of:
- Apple, Intel and Chenbro, for creating the hardware
- Canonical, for creating and maintaining Ubuntu
- The fine devs that created HandBrake
- The guy behind mkvdts2ac3.sh
- The guy behind the original Episode Renamer
- The guys behind SABnzbd+
- The guys behind SickBeard
- The guys at LogicSupply, for providing me with the PCI riser card (a must if you use a SATA RAID PCI card like I did)
- Ebay, for being THE place to buy this stuff (when not available at LogicSupply)
- Sidi Kahawa, for his updated Netatalk guide
- Matthias Kretschmann (aka. krema), for his extensive Netatalk guide (it's really detailed – be sure to read the comments!)
- And finally, Drobo, for the inspiration