Home           RSS           Search

March 14, 2014





Youtube Downloader Wget Perl Script

download videos and even convert them to mp3 or ogg



Why use wget to download YouTube videos ?

Normally, when you want to watch a Youtube video you have to use Adobe Flash in a browser like Firefox, Chrome (Chromium), Opera or Safari. Adobe Flash is incredibly inefficient, a huge security hole and notoriously buggy. Another issue is youtube's constant slow buffering speeds. It is frustrating if you are constantly waiting for the video to buffer in an interactive session.

An alternative is to use a script to download your videos for viewing later. Using wget may or may not be faster then flash in the browser depending on the youtube mirror you connect to. So, the idea is to pre-download a bunch of videos and when you are ready to watch them, they play at the highest quality with no youtube buffer pausing at all. We like to watch videos using VLC which uses less CPU time then flash and can play videos at higher than 1x speeds. In fact, we watch almost all of our vids at 2x (use the + key on the keypad on the right).

We also prefer a simpler scripted solution which does not rely on too many dependencies or proprietary code. For example, youtube-dl requires Python libraries and others require Javascript, Greesemonkey or even PHP to work in Firefox. Simple is better to audit and modify as needed.



Latest Version: 0.42



Advantages of a scripted solution

AdBlock Plus and NoScript safe, No Adobe Flash needed: To download the Youtube video just cut and paste the URL from the browser's URL bar. You do not need to start the video or anything. In fact, if you have NoScript you do not even have to allow java script on the Youtube site. You just need the link and paste it after the script. Simple and easy.

Unlimited download speed: There are no download rate limits with the URL we extract from the HTML page. The videos will download as fast as the Youtube cache server will send the data. The speed primarily depends on the popularity of the video; the most asked for videos are put on the highest bandwidth cache servers. Using a testing server on Linode (Xen VPS Hosting) we easily saw 24 megabytes per second downloads. The maximum upload and download limits on a Linode server is 45 megabytes per second. BTW, Linode gets a 10 out of 10 for overall quality in our opinion. On a side note, we have not found a reliable way of choosing the fastest mirror server. If your ISP has mirror servers they are normally slower then Google so we suggest finding their ips and blocking the ISP mirrors.

No Advertisements: The script will _not_ download any ads or advertisements like what you would see if you used the browser to watch flash videos. The reason is the ads are not part of the video, but a flash overlay Youtube inserts. We do not do anything to remove the ads at all. When you see an ad on Youtube the video is actually paused and the advertisement is overlaid on top of the video window. When the ad finishes the overlay is removed and video is unpaused.



Getting Started

The youtube download script is written in Perl and can be run on Linux, Mac OSX, OpenBSD, FreeBSD or any operating system supporting perl. Since we are using perl the UNIX shell you use, like bash, tcsh, csh or sh does not matter. There is only one dependency you need to have installed on your system; the system binary "wget" and it needs to be in your shell's default path. Wget is incredibly useful not only for this perl script, but for any scripting you may do on the future. BTW, the standard wget package will allow you to download videos using HTTP or HTTPS. To make it easy we included the following lines to install these packages using your OS's package manager.

## Ubuntu Linux
apt-get install wget

## FreeBSD
pkg_add -r wget
   --OR--
pkg install wget
   --OR--
portmaster ftp/wget

## OpenBSD 
pkg_add -i wget


The Youtube Download Perl script

To use the script, copy and paste the block of perl code from the following text box to a file. We are going to call the script youtube_wget.pl for this example, but you can name it anything you like. Remember to make the file executable too, "chmod 755 youtube_wget.pl".

#!/usr/bin/perl -T

use strict;
use warnings;

#
##  Calomel.org  ,:,  Download Youtube videos and music using wget
##    Script Name : youtube_wget_video.pl
##    Version     : 0.42
##    Valid from  : March 2014
##    URL Page    : https://calomel.org/youtube_wget.html
##    OS Support  : Linux, Mac OSX, OpenBSD, FreeBSD or any system with perl
#                `:`
## Two arguments
##    $1 Youtube URL from the browser
##    $2 prefix to the file name of the video (optional)
#

############  options  ##########################################

# Option: what file type do you want to download? The string is used to search
# in the youtube URL so you can choose mp4, webm, avi or flv.  mp4 seems to
# work on the most players like android, ipod, ipad, iphones, vlc and mplayer.
my $fileType = "mp4";

# Option: what visual resolution or quality do you want to download? List
# multiple values just in case the highest quality video is not available, the
# script will look for the next resolution. You can choose "highres" for 4k,
# "hd1080" for 1080p, "hd720" for 720p, "itag=18" which means standard
# definition 640x380 and "itag=17" which is mobile resolution 144p (176x144).
# The script will always prefer to download the highest resolution video format
# from the list if available.
my $resolution = "hd720,itag=18";

# Option: How many times should the script retry the download if wget fails for
# any reason? Do not make this too high as a reoccurring error will just hit
# youtube over and over again. 
my $retryTimes = 20;

# Option: do you want the resolution of the video in the file name? zero(0) is
# no and one(1) is yes. This option simply puts "_hd1080.mp4" or similar at the
# end of the file name.
my $resolutionFilename = 0;

# Option: Force all communication with YouTube to use SSL (https) links. The
# script will simply convert all URL's you pass to the script to use https
# instead of http. Encryption better protects your privacy and may help avoid
# ISP rate limiting. 
my $forceSSL = 1;

# Option: turn on DEBUG mode. Use this to reverse engineering this code if you are
# making changes or you are building your own youtube download script.
my $DEBUG=0;

#################################################################

# initialize retry loop and resolution variables
$ENV{PATH} = "/bin:/usr/bin:/usr/local/bin";
my $prefix = "";
my $retry = 1;
my $retryCounter = 0;
my $resFile = "unknown";
my $user_url = "";
my $user_prefix = "";

# collect the URL from the command line argument
chomp($user_url = $ARGV[0]);
my $url = "$1" if ($user_url =~ m/^([a-zA-Z0-9\_\-\&\?\=\:\.\/]+)$/ or die "\nError: Illegal characters in YouTube URL\n\n" );

# declare the user defined file name prefix if specified
if (defined($ARGV[1])) {
   chomp($user_prefix = $ARGV[1]);
   $prefix = "$1" if ($user_prefix =~ m/^([a-zA-Z0-9\_\-\.\ ]+)$/ or die "\nError: Illegal characters in filename prefix\n\n" );
}

# while loop to retry downloading the video if the script fails for any reason
while ( $retry != 0 && $retryCounter < $retryTimes ) {

# Force SSL (https) download of the html page
$url =~ s/http:\/\//https:\/\//gi if ($forceSSL == 1);

# download the html from the youtube page containing the page title and video
# url. The page title will be used for the local video file name and the url
# will be sanitized and passed to wget for the download.
my $html = `wget -4Ncq --convert-links=off --no-cookies --timeout=20 --user-agent='' --no-check-certificate "$url" -O-`  or die  "\nThere was a problem downloading the HTML page.\n\n";

# format the title of the page to use as the file name
my ($title) = $html =~ m/<title>(.+)<\/title>/si;
$title =~ s/[^\w\d]+/_/g or die "\nError: we could not find the title of the HTML page. Check the URL.\n\n";
$title =~ s/_youtube//ig;
$title =~ s/^_//ig;
$title = lc ($title);
$title =~ s/_amp//ig;

# filter the URL of the video from the HTML page
my ($download) = $html =~ /"url_encoded_fmt_stream_map"(.*)/ig;

# Print all of the separated strings in the HTML page
#print "\n$download\n\n" if ($DEBUG == 1);

# This is where we look through the HTML code and select the file type and
# video quality. 
my @urls = split(',', $download);
OUTERLOOP:
foreach my $val (@urls) {
#   print "\n$val\n\n";

    if ( $val =~ /$fileType/ ) {
       my @res = split(',', $resolution);
       foreach my $ress (@res) {
         if ( $val =~ /$ress/ ) {
         print "\n  html to url seperation complete.\n\n" if ($DEBUG == 1);
         print "$val\n" if ($DEBUG == 1);
         $resFile = $ress;
         $resFile = "sd640" if ( $ress =~ /itag=18/ );
         $resFile = "mobil176" if ( $ress =~ /itag=17/ );
         $download = $val;
         last OUTERLOOP;
         }
       }
    }
}

# clean up the url by translating unicode and removing unwanted strings
print "\n  Re-formatting url for wget...\n\n" if ($DEBUG == 1);
$download =~ s/\:\ \"//;
$download =~ s/%3A/:/g;
$download =~ s/%2F/\//g;
$download =~ s/%3F/\?/g;
$download =~ s/%3D/\=/g;
$download =~ s/%252C/%2C/g;
$download =~ s/%26/\&/g;
$download =~ s/sig=/signature=/g;
$download =~ s/\\u0026/\&/g;
$download =~ s/(type=[^&]+)//g;
$download =~ s/(fallback_host=[^&]+)//g;
$download =~ s/(quality=[^&]+)//g;

# clean up the url 
my ($youtubeurl) = $download =~ /(http?:.+)/;

# url title additon
my ($titleurl) = $html =~ m/<title>(.+)<\/title>/si;
$titleurl =~ s/ - YouTube//ig;
$titleurl =~ s/ /%20/ig;

# combine the youtube url and title string
$download = "$youtubeurl\&title=$titleurl";

# a bit more cleanup as youtube 
#$download =~ s/&+/&/g;
#$download =~ s/&itag=\d+&signature=/&signature=/g;

# combine file variables into the full file name
my $filename = "unknown";
if ( $resolutionFilename == 1 ) {
   $filename = "$prefix$title\_$resFile.$fileType";
  } else {
   $filename = "$prefix$title.$fileType";
}

# Process check: Are we currently downloading this exact same video? Two of the
# same wget processes will overwrite themselves and corrupt the video.
my $running = `ps auwww | grep [w]get | grep -c "$filename"`;
print "\n  Is the same file already being downloaded? $running\n" if ($DEBUG == 1);
if ($running >= 1)
  {
   print "\n  Already $running process, exiting." if ($DEBUG == 1);
   exit 0;
  };

# Force SSL (https) download of the video file.
$download =~ s/http:\/\//https:\/\//g if ($forceSSL == 1);

# Print the long, sanitized youtube url for testing and debugging
print "\n  The following url will be passed to wget:\n\n" if ($DEBUG == 1);
print "\n$download\n" if ($DEBUG == 1);

# print the file name of the video being downloaded for the user 
print "\n Download: $filename\n\n" if ($retryCounter < 1);

# Background the script before wget starts downloading. Use "ps" if you need to
# look for the process running or use "ls -al" to look at the file size and
# date.
fork and exit;

# Download the video 
system("wget", "-4Ncq", "--convert-links=off", "--no-cookies", "--timeout=20", "--no-check-certificate", "--user-agent=''" , "$download", "-O", "$filename");

# Print the error code of wget
print "\n  wget error code: $?\n" if ($DEBUG == 1);

# Exit Status: Check if the file exists and we received the correct error code
# from wget system call. If the download experienced any problems the script
# will run again and try continue the download until the retryTimes count limit
# is reached.

if( $? == 0 && -e "$filename" && ! -z "$filename" )
   {
      print "\n  Finished: $filename\n\n" if ($DEBUG == 1);
    # print "\n  Success: $filename\n\n";
      $retry = 0;
   }
 else
   {
      print STDERR "\n  FAILED: $filename\n\n" if ($DEBUG == 1);
    # print "\n  FAILED: $filename\n\n";
      $retry = 1;
      $retryCounter++;
    # sleep $retryCounter;
      sleep 1;
   }
}

#### EOF #####


How do I use the script ?

Once you have the script setup you just need to find a Youtube video. We chose a video from Tobygames as he meets his first Giant Radscorpion in Fallout New Vegas. Execute the script with the youtube URL copy and pasted from Firefox's URL bar. Make note you can add one more argument to the end of the command line to add a prefix to the file name. Here is an example of both options; notice the change in files names as the second example has "toby_" as the file name prefix. Also note some of the URLS through youtube have ampersands "&" in them. For these types of URL's just use double quotes around the url so your shell passes the full string into the script.

## Example 1: Here we just pass the youtube URL
#
user@machine$ ./youtube_wget.pl "https://www.youtube.com/watch?v=ejkm5uGoxs4"

   Download: radscorpion.mp4


## Example 2: Here we pass the Youtube URL and the file name prefix "toby_"
#
user@machine$ ./youtube_wget.pl "https://www.youtube.com/watch?v=ejkm5uGoxs4" toby_

   Download: toby_radscorpion.mp4

The video will download in the background and save to your current directory. You can play it with your favorite video player, we prefer VLC for example.



Things to keep in mind...

File name is the same as the name of the web page: Notice the file name is the same as the title of the Youtube web page. We have also scrubbed the title to take out all special characters and reduce all letters to lower case. This makes it easier read and to run on the command line.

Save Location: The video will be saved in your current directory.

Script methodology: The wget line will run in the background. You can start as many of these downloads as you want. We have started as many as a dozen simultaneous downloads without issue. The script will finish silently; meaning when the download is finished you will not get any notification.

Wget process state: You can check if the download is running by looking at the process list (ps) and grep'ing for wget. Something like, ps -aux | grep wget will work. At this point there is no way to tell how fast the download is going. What you can do is look at the file size change using ls -la and estimate from there. You can start watching the video file right away too. The file is downloaded serially, so as soon as the file starts downloading you should be able to start VLC if want to watch the video right away.

Video file type: The video will download in the file type you specify. mp4 seems to be the most compatible type, but WebM which is also called VP8 and AVI are available. WebM is a digital multimedia container file format promoted by the open-source WebM Project headed by Google. It comprises a subset of the Matroska multimedia container format. If you current media player does not support webm then you need a codec for your OS. Just search on Google for "webm codec" and you should get pointed in the right direction. Note, you can play this format with the VLC media player which is available on all OS's. VLC is a free and open source cross-platform multimedia player and framework that plays most multimedia files as well as DVD, Audio CD, VCD, and various streaming protocols. VLC can also play the videos at greater than 1x speed by hitting the plus "+" key on the keypad. When playing videos at anything faster than 1x the sound will be automatically pitch corrected. For example, we like to watch Quill18, Day9 and Sacriel42 videos at 2x.

Always use the latest script version: Youtube changes the format of their HTML pages every once in a while which consequently breaks download scripts like what we have here. The average amount of time between HTML format changes is three(3) months. If you find this script no longer works make sure to check back on this page for any updates. We will do our best to keep this command line option working since we use this script at least once a day. Make note, at the top of the script we have the version number and date the script is good from. We will also post on the RSS feed (link at the top of the page) when a new version is available.



How do I make an audio mp3 or ogg from a youtube video ?

At some point you will want to save off the sound from a video. A good case is downloading an instructional video and listening to it on your music player like a Sansa Clip, iPad, iPhone or iPod. We like to download class videos from the Massachusetts Institute of Technology (MIT) and listen to them in the car.

You can use the above download script to get a video and convert the video's soundtrack to MP3 (or OGG or any other) format using avconv. You will need to install avconv in order to extract the audio. For Ubuntu use "apt-get install libav-tools". OpenBSD "pkg_add -i ffmpeg" and FreeBSD use "pkg_add -r ffmpeg" instead if avconv is not available as a package. For Mac OSX you may want to look at ffmpegX.

## Convert the audio from a Youtube video to mp3 or ogg, audio only.

## download the video. (Same link to Tobygames as above)
user@machine$ ./youtube_wget.pl https://www.youtube.com/watch?v=ejkm5uGoxs4
  Downloading:  radscorpion.flv

## Convert video to mp3, audio only
user@machine$ avconv -i radscorpion.flv -vn -ab 128 toby_audio.mp3

## Convert video to ogg, audio only
user@machine$ avconv -i radscorpion.flv -vn -ab 128 toby_audio.ogg



Questions?

How can I download Youtube Playlists ?

We suggest using the perl script, YTPlaylistFetcher by Kai. Simply download the script and make it executable. Then run the script with "-f url" followed by the youtube playlist URL. The script will collect all of the playlist urls in the text file "url". You can then use our download script in a while loop to download all the videos. Works perfectly.

# collect the urls from the youtube playlist. For example, SSoHPKC's well 
# done walkthrough of Fallout New Vegas consisting of 116 videos.
./ytplaylistfetcher -f url "https://www.youtube.com/playlist?list=PL59360CCC81082296"

# now download the videos with our script including a 30 second pause between
# requests to reduce load on youtube's servers as well as our local I/O.
while read input; do ./youtube_wget.pl $input;sleep 30; done < url

How about an Android device app ?

Yes, if you have an Android phone or tablet then check out the Android App - YouTube Downloader . You can even choose the format of the video from mp4, to avi to webm! Note: the app is not in the Google store yet, but you can download it directly from the sourceforge link we provided.

Is there a GUI or "clickable" download tool I can use ?

The best Firefox add-on we have found is called Flash Video Downloader - Youtube Downloader by Pavel Shcherbakov . It is incredibly easy to use and offers multiple download formats and a choice of low, medium, high or HD quality. We use this add on all the time when it is easier to use Firefox then our perl script.

This script only supports youtube, are you going to support other sites?

No, we only use youtube at this time. For download support of other sites you should check out youtube-dl. youtube-dl supports almost 20 different video sites including youtube. You might even be able to install youtube-dl using apt-get or yum.

I have a patch or the script is broke

You are welcome to mail us. Please make sure you look at any errors the script outputs to see if you can see the cause of the error. The contact link is at the bottom of this page.





Questions, comments, or suggestions? Contact Calomel.org or