Merge pull request #9 from rg3/master

up
2025-03-13 07:09:58 +08:00 · 2017-09-01 12:50:57 +08:00 · 2017-09-01 12:50:57 +08:00 · 88dcfd3120
commit 88dcfd3120
parent ebd8bce48f 7998520933
28 changed files with 699 additions and 288 deletions
--- a/.github/ISSUE_TEMPLATE.md
+++ b/.github/ISSUE_TEMPLATE.md
@ -6,8 +6,8 @@
 ---
-### Make sure you are using the *latest* version: run `youtube-dl --version` and ensure your version is *2017.08.18*. If it's not, read [this FAQ entry](https://github.com/rg3/youtube-dl/blob/master/README.md#how-do-i-update-youtube-dl) and update. Issues with outdated version will be rejected.
+### Make sure you are using the *latest* version: run `youtube-dl --version` and ensure your version is *2017.08.27.1*. If it's not, read [this FAQ entry](https://github.com/rg3/youtube-dl/blob/master/README.md#how-do-i-update-youtube-dl) and update. Issues with outdated version will be rejected.
- [ ] I've **verified** and **I assure** that I'm running youtube-dl **2017.08.18**
+- [ ] I've **verified** and **I assure** that I'm running youtube-dl **2017.08.27.1**
 ### Before submitting an *issue* make sure you have:
 - [ ] At least skimmed through the [README](https://github.com/rg3/youtube-dl/blob/master/README.md), **most notably** the [FAQ](https://github.com/rg3/youtube-dl#faq) and [BUGS](https://github.com/rg3/youtube-dl#bugs) sections
@ -35,7 +35,7 @@ Add the `-v` flag to **your command line** you run youtube-dl with (`youtube-dl
 [debug] User config: []
 [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
 [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
-[debug] youtube-dl version 2017.08.18
+[debug] youtube-dl version 2017.08.27.1
 [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
 [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
 [debug] Proxy map: {}
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@ -3,7 +3,7 @@
 $ youtube-dl -v <your command line>
 [debug] System config: []
 [debug] User config: []
-[debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
+[debug] Command-line args: [u'-v', u'https://www.youtube.com/watch?v=BaW_jenozKcj']
 [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
 [debug] youtube-dl version 2015.12.06
 [debug] Git HEAD: 135392e
@ -34,7 +34,7 @@ For bug reports, this means that your report should contain the *complete* outpu
 If your server has multiple IPs or you suspect censorship, adding `--call-home` may be a good idea to get more diagnostics. If the error is `ERROR: Unable to extract ...` and you cannot reproduce it from multiple countries, add `--dump-pages` (warning: this will yield a rather large output, redirect it to the file `log.txt` by adding `>log.txt 2>&1` to your command-line) or upload the `.dump` files you get when you add `--write-pages` [somewhere](https://gist.github.com/).
-**Site support requests must contain an example URL**. An example URL is a URL you might want to download, like `http://www.youtube.com/watch?v=BaW_jenozKc`. There should be an obvious video present. Except under very special circumstances, the main page of a video service (e.g. `http://www.youtube.com/`) is *not* an example URL.
+**Site support requests must contain an example URL**. An example URL is a URL you might want to download, like `https://www.youtube.com/watch?v=BaW_jenozKc`. There should be an obvious video present. Except under very special circumstances, the main page of a video service (e.g. `https://www.youtube.com/`) is *not* an example URL.
 ###  Are you using the latest version?
@ -70,7 +70,7 @@ It may sound strange, but some bug reports we receive are completely unrelated t
 # DEVELOPER INSTRUCTIONS
-Most users do not need to build youtube-dl and can [download the builds](http://rg3.github.io/youtube-dl/download.html) or get them from their distribution.
+Most users do not need to build youtube-dl and can [download the builds](https://rg3.github.io/youtube-dl/download.html) or get them from their distribution.
 To run youtube-dl as a developer, you don't need to build anything either. Simply execute
@ -118,7 +118,7 @@ After you have ensured this site is distributing its content legally, you can fo
    class YourExtractorIE(InfoExtractor):
        _VALID_URL = r'https?://(?:www\.)?yourextractor\.com/watch/(?P<id>[0-9]+)'
        _TEST = {
-            'url': 'http://yourextractor.com/watch/42',
+            'url': 'https://yourextractor.com/watch/42',
            'md5': 'TODO: md5 sum of the first 10241 bytes of the video file (use --test)',
            'info_dict': {
                'id': '42',
@ -151,8 +151,8 @@ After you have ensured this site is distributing its content legally, you can fo
 5. Add an import in [`youtube_dl/extractor/extractors.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/extractors.py).
 6. Run `python test/test_download.py TestDownload.test_YourExtractor`. This *should fail* at first, but you can continually re-run it until you're done. If you decide to add more than one test, then rename ``_TEST`` to ``_TESTS`` and make it into a list of dictionaries. The tests will then be named `TestDownload.test_YourExtractor`, `TestDownload.test_YourExtractor_1`, `TestDownload.test_YourExtractor_2`, etc.
 7. Have a look at [`youtube_dl/extractor/common.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py) for possible helper methods and a [detailed description of what your extractor should and may return](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py#L74-L252). Add tests and code for as many as you want.
-8. Make sure your code follows [youtube-dl coding conventions](#youtube-dl-coding-conventions) and check the code with [flake8](https://pypi.python.org/pypi/flake8). Also make sure your code works under all [Python](http://www.python.org/) versions claimed supported by youtube-dl, namely 2.6, 2.7, and 3.2+.
+8. Make sure your code follows [youtube-dl coding conventions](#youtube-dl-coding-conventions) and check the code with [flake8](https://pypi.python.org/pypi/flake8). Also make sure your code works under all [Python](https://www.python.org/) versions claimed supported by youtube-dl, namely 2.6, 2.7, and 3.2+.
-9. When the tests pass, [add](http://git-scm.com/docs/git-add) the new files and [commit](http://git-scm.com/docs/git-commit) them and [push](http://git-scm.com/docs/git-push) the result, like this:
+9. When the tests pass, [add](https://git-scm.com/docs/git-add) the new files and [commit](https://git-scm.com/docs/git-commit) them and [push](https://git-scm.com/docs/git-push) the result, like this:
        $ git add youtube_dl/extractor/extractors.py
        $ git add youtube_dl/extractor/yourextractor.py
--- a/51
+++ b/51
@ -1,3 +1,52 @@
 version 2017.08.27.1
 Extractors
 * [youtube] Fix extraction with --youtube-skip-dash-manifest enabled (#14037)
 version 2017.08.27
 Core
 + [extractor/common] Extract height and format id for HTML5 videos (#14034)
 * [downloader/http] Rework HTTP downloader (#506, #809, #2849, #4240, #6023,
  #8625, #9483)
    * Simplify code and split into separate routines to facilitate maintaining
    * Make retry mechanism work on errors during actual download not only
      during connection establishment phase
    * Retry on ECONNRESET and ETIMEDOUT during reading data from network
    * Retry on content too short
    * Show error description on retry
 Extractors
 * [generic] Lower preference for extraction from LD-JSON
 * [rai] Fix audio formats extraction (#14024)
 * [youtube] Fix controversy videos extraction (#14027, #14029)
 * [mixcloud] Fix extraction (#14015, #14020)
 version 2017.08.23
 Core
 + [extractor/common] Introduce _parse_xml
 * [extractor/common] Make HLS and DASH extraction in_parse_html5_media_entries
  non fatal (#13970)
 * [utils] Fix unescapeHTML for misformed string like "&a&quot;" (#13935)
 Extractors
 * [cbc:watch] Bypass geo restriction (#13993)
 * [toutv] Relax DRM check (#13994)
 + [googledrive] Add support for subtitles (#13619, #13638)
 * [pornhub] Relax uploader regular expression (#13906, #13975)
 * [bandcamp:album] Extract track titles (#13962)
 + [bbccouk] Add support for events URLs (#13893)
 + [liveleak] Support multi-video pages (#6542)
 + [liveleak] Support another liveleak embedding pattern (#13336)
 * [cda] Fix extraction (#13935)
 + [laola1tv] Add support for tv.ittf.com (#13965)
 * [mixcloud] Fix extraction (#13958, #13974, #13980, #14003)
 version 2017.08.18
 Core
@ -118,7 +167,7 @@ Extractors
 * [youku:show] Fix playlist extraction (#13248)
 + [dispeak] Recognize sevt subdomain (#13276)
 * [adn] Improve error reporting (#13663)
-* [crunchyroll] Relax series and season regex (#13659)
+* [crunchyroll] Relax series and season regular expression (#13659)
 + [spiegel:article] Add support for nexx iframe embeds (#13029)
 + [nexx:embed] Add support for iframe embeds
 * [nexx] Improve JS embed extraction
--- a/11
+++ b/11
@ -46,8 +46,15 @@ tar: youtube-dl.tar.gz
 pypi-files: youtube-dl.bash-completion README.txt youtube-dl.1 youtube-dl.fish
 youtube-dl: youtube_dl/*.py youtube_dl/*/*.py
-	zip --quiet youtube-dl youtube_dl/*.py youtube_dl/*/*.py
+	mkdir -p zip
-	zip --quiet --junk-paths youtube-dl youtube_dl/__main__.py
+	for d in youtube_dl youtube_dl/downloader youtube_dl/extractor youtube_dl/postprocessor ; do \
 	  mkdir -p zip/$$d ;\
 	  cp -pPR $$d/*.py zip/$$d/ ;\
 	done
 	touch -t 200001010101 zip/youtube_dl/*.py zip/youtube_dl/*/*.py
 	mv zip/youtube_dl/__main__.py zip/
 	cd zip ; zip -q ../youtube-dl youtube_dl/*.py youtube_dl/*/*.py __main__.py
 	rm -rf zip
 	echo '#!$(PYTHON)' > youtube-dl
 	cat youtube-dl.zip >> youtube-dl
 	rm youtube-dl.zip
--- a/README.md
+++ b/README.md
@ -25,7 +25,7 @@ If you do not have curl, you can alternatively use a recent wget:
    sudo wget https://yt-dl.org/downloads/latest/youtube-dl -O /usr/local/bin/youtube-dl
    sudo chmod a+rx /usr/local/bin/youtube-dl
-Windows users can [download an .exe file](https://yt-dl.org/latest/youtube-dl.exe) and place it in any location on their [PATH](http://en.wikipedia.org/wiki/PATH_%28variable%29) except for `%SYSTEMROOT%\System32` (e.g. **do not** put in `C:\Windows\System32`).
+Windows users can [download an .exe file](https://yt-dl.org/latest/youtube-dl.exe) and place it in any location on their [PATH](https://en.wikipedia.org/wiki/PATH_%28variable%29) except for `%SYSTEMROOT%\System32` (e.g. **do not** put in `C:\Windows\System32`).
 You can also use pip:
@ -33,7 +33,7 @@ You can also use pip:
 This command will update youtube-dl if you have already installed it. See the [pypi page](https://pypi.python.org/pypi/youtube_dl) for more information.
-OS X users can install youtube-dl with [Homebrew](http://brew.sh/):
+OS X users can install youtube-dl with [Homebrew](https://brew.sh/):
    brew install youtube-dl
@ -458,7 +458,7 @@ You can also use `--config-location` if you want to use custom configuration fil
 ### Authentication with `.netrc` file
-You may also want to configure automatic credentials storage for extractors that support authentication (by providing login and password with `--username` and `--password`) in order not to pass credentials as command line arguments on every youtube-dl execution and prevent tracking plain text passwords in the shell command history. You can achieve this using a [`.netrc` file](http://stackoverflow.com/tags/.netrc/info) on a per extractor basis. For that you will need to create a `.netrc` file in your `$HOME` and restrict permissions to read/write by only you:
+You may also want to configure automatic credentials storage for extractors that support authentication (by providing login and password with `--username` and `--password`) in order not to pass credentials as command line arguments on every youtube-dl execution and prevent tracking plain text passwords in the shell command history. You can achieve this using a [`.netrc` file](https://stackoverflow.com/tags/.netrc/info) on a per extractor basis. For that you will need to create a `.netrc` file in your `$HOME` and restrict permissions to read/write by only you:
 ```
 touch $HOME/.netrc
 chmod a-rwx,u+rw $HOME/.netrc
@ -485,7 +485,7 @@ The `-o` option allows users to indicate a template for the output file names.
 **tl;dr:** [navigate me to examples](#output-template-examples).
-The basic usage is not to set any template arguments when downloading a single file, like in `youtube-dl -o funny_video.flv "http://some/video"`. However, it may contain special sequences that will be replaced when downloading each video. The special sequences may be formatted according to [python string formatting operations](https://docs.python.org/2/library/stdtypes.html#string-formatting). For example, `%(NAME)s` or `%(NAME)05d`. To clarify, that is a percent symbol followed by a name in parentheses, followed by a formatting operations. Allowed names along with sequence type are:
+The basic usage is not to set any template arguments when downloading a single file, like in `youtube-dl -o funny_video.flv "https://some/video"`. However, it may contain special sequences that will be replaced when downloading each video. The special sequences may be formatted according to [python string formatting operations](https://docs.python.org/2/library/stdtypes.html#string-formatting). For example, `%(NAME)s` or `%(NAME)05d`. To clarify, that is a percent symbol followed by a name in parentheses, followed by a formatting operations. Allowed names along with sequence type are:
 - `id` (string): Video identifier
 - `title` (string): Video title
@ -603,7 +603,7 @@ $ youtube-dl -o '%(uploader)s/%(playlist)s/%(playlist_index)s - %(title)s.%(ext)
 $ youtube-dl -u user -p password -o '~/MyVideos/%(playlist)s/%(chapter_number)s - %(chapter)s/%(title)s.%(ext)s' https://www.udemy.com/java-tutorial/
 # Download entire series season keeping each series and each season in separate directory under C:/MyVideos
-$ youtube-dl -o "C:/MyVideos/%(series)s/%(season_number)s - %(season)s/%(episode_number)s - %(episode)s.%(ext)s" http://videomore.ru/kino_v_detalayah/5_sezon/367617
+$ youtube-dl -o "C:/MyVideos/%(series)s/%(season_number)s - %(season)s/%(episode_number)s - %(episode)s.%(ext)s" https://videomore.ru/kino_v_detalayah/5_sezon/367617
 # Stream the video being downloaded to stdout
 $ youtube-dl -o - BaW_jenozKc
@ -716,17 +716,17 @@ $ youtube-dl --dateafter 20000101 --datebefore 20091231
 ### How do I update youtube-dl?
-If you've followed [our manual installation instructions](http://rg3.github.io/youtube-dl/download.html), you can simply run `youtube-dl -U` (or, on Linux, `sudo youtube-dl -U`).
+If you've followed [our manual installation instructions](https://rg3.github.io/youtube-dl/download.html), you can simply run `youtube-dl -U` (or, on Linux, `sudo youtube-dl -U`).
 If you have used pip, a simple `sudo pip install -U youtube-dl` is sufficient to update.
-If you have installed youtube-dl using a package manager like *apt-get* or *yum*, use the standard system update mechanism to update. Note that distribution packages are often outdated. As a rule of thumb, youtube-dl releases at least once a month, and often weekly or even daily. Simply go to http://yt-dl.org/ to find out the current version. Unfortunately, there is nothing we youtube-dl developers can do if your distribution serves a really outdated version. You can (and should) complain to your distribution in their bugtracker or support forum.
+If you have installed youtube-dl using a package manager like *apt-get* or *yum*, use the standard system update mechanism to update. Note that distribution packages are often outdated. As a rule of thumb, youtube-dl releases at least once a month, and often weekly or even daily. Simply go to https://yt-dl.org to find out the current version. Unfortunately, there is nothing we youtube-dl developers can do if your distribution serves a really outdated version. You can (and should) complain to your distribution in their bugtracker or support forum.
 As a last resort, you can also uninstall the version installed by your package manager and follow our manual installation instructions. For that, remove the distribution's package, with a line like
    sudo apt-get remove -y youtube-dl
-Afterwards, simply follow [our manual installation instructions](http://rg3.github.io/youtube-dl/download.html):
+Afterwards, simply follow [our manual installation instructions](https://rg3.github.io/youtube-dl/download.html):
 ```
 sudo wget https://yt-dl.org/latest/youtube-dl -O /usr/local/bin/youtube-dl
@ -766,11 +766,11 @@ Apparently YouTube requires you to pass a CAPTCHA test if you download too much.
 youtube-dl works fine on its own on most sites. However, if you want to convert video/audio, you'll need [avconv](https://libav.org/) or [ffmpeg](https://www.ffmpeg.org/). On some sites - most notably YouTube - videos can be retrieved in a higher quality format without sound. youtube-dl will detect whether avconv/ffmpeg is present and automatically pick the best option.
-Videos or video formats streamed via RTMP protocol can only be downloaded when [rtmpdump](https://rtmpdump.mplayerhq.hu/) is installed. Downloading MMS and RTSP videos requires either [mplayer](http://mplayerhq.hu/) or [mpv](https://mpv.io/) to be installed.
+Videos or video formats streamed via RTMP protocol can only be downloaded when [rtmpdump](https://rtmpdump.mplayerhq.hu/) is installed. Downloading MMS and RTSP videos requires either [mplayer](https://mplayerhq.hu/) or [mpv](https://mpv.io/) to be installed.
 ### I have downloaded a video but how can I play it?
-Once the video is fully downloaded, use any video player, such as [mpv](https://mpv.io/), [vlc](http://www.videolan.org/) or [mplayer](http://www.mplayerhq.hu/).
+Once the video is fully downloaded, use any video player, such as [mpv](https://mpv.io/), [vlc](https://www.videolan.org/) or [mplayer](https://www.mplayerhq.hu/).
 ### I extracted a video URL with `-g`, but it does not play on another machine / in my web browser.
@ -845,10 +845,10 @@ Use the `-o` to specify an [output template](#output-template), for example `-o
 ### How do I download a video starting with a `-`?
-Either prepend `http://www.youtube.com/watch?v=` or separate the ID from the options with `--`:
+Either prepend `https://www.youtube.com/watch?v=` or separate the ID from the options with `--`:
    youtube-dl -- -wNyEUrxzFU
-    youtube-dl "http://www.youtube.com/watch?v=-wNyEUrxzFU"
+    youtube-dl "https://www.youtube.com/watch?v=-wNyEUrxzFU"
 ### How do I pass cookies to youtube-dl?
@ -862,9 +862,9 @@ Passing cookies to youtube-dl is a good way to workaround login when a particula
 ### How do I stream directly to media player?
-You will first need to tell youtube-dl to stream media to stdout with `-o -`, and also tell your media player to read from stdin (it must be capable of this for streaming) and then pipe former to latter. For example, streaming to [vlc](http://www.videolan.org/) can be achieved with:
+You will first need to tell youtube-dl to stream media to stdout with `-o -`, and also tell your media player to read from stdin (it must be capable of this for streaming) and then pipe former to latter. For example, streaming to [vlc](https://www.videolan.org/) can be achieved with:
-    youtube-dl -o - "http://www.youtube.com/watch?v=BaW_jenozKcj" | vlc -
+    youtube-dl -o - "https://www.youtube.com/watch?v=BaW_jenozKcj" | vlc -
 ### How do I download only new videos from a playlist?
@ -884,7 +884,7 @@ When youtube-dl detects an HLS video, it can download it either with the built-i
 When youtube-dl knows that one particular downloader works better for a given website, that downloader will be picked. Otherwise, youtube-dl will pick the best downloader for general compatibility, which at the moment happens to be ffmpeg. This choice may change in future versions of youtube-dl, with improvements of the built-in downloader and/or ffmpeg.
-In particular, the generic extractor (used when your website is not in the [list of supported sites by youtube-dl](http://rg3.github.io/youtube-dl/supportedsites.html) cannot mandate one specific downloader.
+In particular, the generic extractor (used when your website is not in the [list of supported sites by youtube-dl](https://rg3.github.io/youtube-dl/supportedsites.html) cannot mandate one specific downloader.
 If you put either `--hls-prefer-native` or `--hls-prefer-ffmpeg` into your configuration, a different subset of videos will fail to download correctly. Instead, it is much better to [file an issue](https://yt-dl.org/bug) or a pull request which details why the native or the ffmpeg HLS downloader is a better choice for your use case.
@ -910,7 +910,7 @@ Feel free to bump the issue from time to time by writing a small comment ("Issue
 ### How can I detect whether a given URL is supported by youtube-dl?
-For one, have a look at the [list of supported sites](docs/supportedsites.md). Note that it can sometimes happen that the site changes its URL scheme (say, from http://example.com/video/1234567 to http://example.com/v/1234567 ) and youtube-dl reports an URL of a service in that list as unsupported. In that case, simply report a bug.
+For one, have a look at the [list of supported sites](docs/supportedsites.md). Note that it can sometimes happen that the site changes its URL scheme (say, from https://example.com/video/1234567 to https://example.com/v/1234567 ) and youtube-dl reports an URL of a service in that list as unsupported. In that case, simply report a bug.
 It is *not* possible to detect whether a URL is supported or not. That's because youtube-dl contains a generic extractor which matches **all** URLs. You may be tempted to disable, exclude, or remove the generic extractor, but the generic extractor not only allows users to extract videos from lots of websites that embed a video from another service, but may also be used to extract video from a service that it's hosting itself. Therefore, we neither recommend nor support disabling, excluding, or removing the generic extractor.
@ -924,7 +924,7 @@ youtube-dl is an open-source project manned by too few volunteers, so we'd rathe
 # DEVELOPER INSTRUCTIONS
-Most users do not need to build youtube-dl and can [download the builds](http://rg3.github.io/youtube-dl/download.html) or get them from their distribution.
+Most users do not need to build youtube-dl and can [download the builds](https://rg3.github.io/youtube-dl/download.html) or get them from their distribution.
 To run youtube-dl as a developer, you don't need to build anything either. Simply execute
@ -972,7 +972,7 @@ After you have ensured this site is distributing its content legally, you can fo
    class YourExtractorIE(InfoExtractor):
        _VALID_URL = r'https?://(?:www\.)?yourextractor\.com/watch/(?P<id>[0-9]+)'
        _TEST = {
-            'url': 'http://yourextractor.com/watch/42',
+            'url': 'https://yourextractor.com/watch/42',
            'md5': 'TODO: md5 sum of the first 10241 bytes of the video file (use --test)',
            'info_dict': {
                'id': '42',
@ -1005,8 +1005,8 @@ After you have ensured this site is distributing its content legally, you can fo
 5. Add an import in [`youtube_dl/extractor/extractors.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/extractors.py).
 6. Run `python test/test_download.py TestDownload.test_YourExtractor`. This *should fail* at first, but you can continually re-run it until you're done. If you decide to add more than one test, then rename ``_TEST`` to ``_TESTS`` and make it into a list of dictionaries. The tests will then be named `TestDownload.test_YourExtractor`, `TestDownload.test_YourExtractor_1`, `TestDownload.test_YourExtractor_2`, etc.
 7. Have a look at [`youtube_dl/extractor/common.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py) for possible helper methods and a [detailed description of what your extractor should and may return](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py#L74-L252). Add tests and code for as many as you want.
-8. Make sure your code follows [youtube-dl coding conventions](#youtube-dl-coding-conventions) and check the code with [flake8](https://pypi.python.org/pypi/flake8). Also make sure your code works under all [Python](http://www.python.org/) versions claimed supported by youtube-dl, namely 2.6, 2.7, and 3.2+.
+8. Make sure your code follows [youtube-dl coding conventions](#youtube-dl-coding-conventions) and check the code with [flake8](https://pypi.python.org/pypi/flake8). Also make sure your code works under all [Python](https://www.python.org/) versions claimed supported by youtube-dl, namely 2.6, 2.7, and 3.2+.
-9. When the tests pass, [add](http://git-scm.com/docs/git-add) the new files and [commit](http://git-scm.com/docs/git-commit) them and [push](http://git-scm.com/docs/git-push) the result, like this:
+9. When the tests pass, [add](https://git-scm.com/docs/git-add) the new files and [commit](https://git-scm.com/docs/git-commit) them and [push](https://git-scm.com/docs/git-push) the result, like this:
        $ git add youtube_dl/extractor/extractors.py
        $ git add youtube_dl/extractor/yourextractor.py
@ -1162,7 +1162,7 @@ import youtube_dl
 ydl_opts = {}
 with youtube_dl.YoutubeDL(ydl_opts) as ydl:
-    ydl.download(['http://www.youtube.com/watch?v=BaW_jenozKc'])
+    ydl.download(['https://www.youtube.com/watch?v=BaW_jenozKc'])
 ```
 Most likely, you'll want to use various options. For a list of options available, have a look at [`youtube_dl/YoutubeDL.py`](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/YoutubeDL.py#L129-L279). For a start, if you want to intercept youtube-dl's output, set a `logger` object.
@ -1201,19 +1201,19 @@ ydl_opts = {
    'progress_hooks': [my_hook],
 }
 with youtube_dl.YoutubeDL(ydl_opts) as ydl:
-    ydl.download(['http://www.youtube.com/watch?v=BaW_jenozKc'])
+    ydl.download(['https://www.youtube.com/watch?v=BaW_jenozKc'])
 ```
 # BUGS
-Bugs and suggestions should be reported at: <https://github.com/rg3/youtube-dl/issues>. Unless you were prompted to or there is another pertinent reason (e.g. GitHub fails to accept the bug report), please do not send bug reports via personal email. For discussions, join us in the IRC channel [#youtube-dl](irc://chat.freenode.net/#youtube-dl) on freenode ([webchat](http://webchat.freenode.net/?randomnick=1&channels=youtube-dl)).
+Bugs and suggestions should be reported at: <https://github.com/rg3/youtube-dl/issues>. Unless you were prompted to or there is another pertinent reason (e.g. GitHub fails to accept the bug report), please do not send bug reports via personal email. For discussions, join us in the IRC channel [#youtube-dl](irc://chat.freenode.net/#youtube-dl) on freenode ([webchat](https://webchat.freenode.net/?randomnick=1&channels=youtube-dl)).
 **Please include the full output of youtube-dl when run with `-v`**, i.e. **add** `-v` flag to **your command line**, copy the **whole** output and post it in the issue body wrapped in \`\`\` for better formatting. It should look similar to this:
 ```
 $ youtube-dl -v <your command line>
 [debug] System config: []
 [debug] User config: []
-[debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
+[debug] Command-line args: [u'-v', u'https://www.youtube.com/watch?v=BaW_jenozKcj']
 [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
 [debug] youtube-dl version 2015.12.06
 [debug] Git HEAD: 135392e
@ -1244,7 +1244,7 @@ For bug reports, this means that your report should contain the *complete* outpu
 If your server has multiple IPs or you suspect censorship, adding `--call-home` may be a good idea to get more diagnostics. If the error is `ERROR: Unable to extract ...` and you cannot reproduce it from multiple countries, add `--dump-pages` (warning: this will yield a rather large output, redirect it to the file `log.txt` by adding `>log.txt 2>&1` to your command-line) or upload the `.dump` files you get when you add `--write-pages` [somewhere](https://gist.github.com/).
-**Site support requests must contain an example URL**. An example URL is a URL you might want to download, like `http://www.youtube.com/watch?v=BaW_jenozKc`. There should be an obvious video present. Except under very special circumstances, the main page of a video service (e.g. `http://www.youtube.com/`) is *not* an example URL.
+**Site support requests must contain an example URL**. An example URL is a URL you might want to download, like `https://www.youtube.com/watch?v=BaW_jenozKc`. There should be an obvious video present. Except under very special circumstances, the main page of a video service (e.g. `https://www.youtube.com/`) is *not* an example URL.
 ###  Are you using the latest version?
--- a/docs/supportedsites.md
+++ b/docs/supportedsites.md
@ -363,6 +363,7 @@
 - **IPrima**
 - **iqiyi**: 爱奇艺
 - **Ir90Tv**
 - **ITTF**
 - **ITV**
 - **ivi**: ivi.ru
 - **ivi:compilation**: ivi.ru compilations
@ -419,6 +420,7 @@
 - **limelight:channel_list**
 - **LiTV**
 - **LiveLeak**
 - **LiveLeakEmbed**
 - **livestream**
 - **livestream:original**
 - **LnkGo**
--- a/test/test_utils.py
+++ b/test/test_utils.py
@ -279,6 +279,7 @@ class TestUtil(unittest.TestCase):
        self.assertEqual(unescapeHTML('&#47;'), '/')
        self.assertEqual(unescapeHTML('&eacute;'), 'é')
        self.assertEqual(unescapeHTML('&#2013266066;'), '&#2013266066;')
        self.assertEqual(unescapeHTML('&a&quot;'), '&a"')
        # HTML5 entities
        self.assertEqual(unescapeHTML('&period;&apos;'), '.\'')
--- a/youtube_dl/downloader/common.py
+++ b/youtube_dl/downloader/common.py
@ -304,11 +304,11 @@ class FileDownloader(object):
        """Report attempt to resume at given byte."""
        self.to_screen('[download] Resuming download at byte %s' % resume_len)
-    def report_retry(self, count, retries):
+    def report_retry(self, err, count, retries):
        """Report retry in case of HTTP error 5xx"""
        self.to_screen(
-            '[download] Got server HTTP error. Retrying (attempt %d of %s)...'
+            '[download] Got server HTTP error: %s. Retrying (attempt %d of %s)...'
-            % (count, self.format_retries(retries)))
+            % (error_to_compat_str(err), count, self.format_retries(retries)))
    def report_file_already_downloaded(self, file_name):
        """Report file has already been fully downloaded."""
--- a/youtube_dl/downloader/http.py
+++ b/youtube_dl/downloader/http.py
@ -22,8 +22,16 @@ from ..utils import (
 class HttpFD(FileDownloader):
    def real_download(self, filename, info_dict):
        url = info_dict['url']
-        tmpfilename = self.temp_name(filename)
+
-        stream = None
+        class DownloadContext(dict):
            __getattr__ = dict.get
            __setattr__ = dict.__setitem__
            __delattr__ = dict.__delitem__
        ctx = DownloadContext()
        ctx.filename = filename
        ctx.tmpfilename = self.temp_name(filename)
        ctx.stream = None
        # Do not include the Accept-Encoding header
        headers = {'Youtubedl-no-compression': 'True'}
@ -38,46 +46,51 @@ class HttpFD(FileDownloader):
        if is_test:
            request.add_header('Range', 'bytes=0-%s' % str(self._TEST_FILE_SIZE - 1))
-        # Establish possible resume length
+        ctx.open_mode = 'wb'
-        if os.path.isfile(encodeFilename(tmpfilename)):
+        ctx.resume_len = 0
            resume_len = os.path.getsize(encodeFilename(tmpfilename))
        else:
            resume_len = 0
        open_mode = 'wb'
        if resume_len != 0:
        if self.params.get('continuedl', True):
-                self.report_resuming_byte(resume_len)
+            # Establish possible resume length
-                request.add_header('Range', 'bytes=%d-' % resume_len)
+            if os.path.isfile(encodeFilename(ctx.tmpfilename)):
-                open_mode = 'ab'
+                ctx.resume_len = os.path.getsize(encodeFilename(ctx.tmpfilename))
            else:
                resume_len = 0
        count = 0
        retries = self.params.get('retries', 0)
-        while count <= retries:
+
        class SucceedDownload(Exception):
            pass
        class RetryDownload(Exception):
            def __init__(self, source_error):
                self.source_error = source_error
        def establish_connection():
            if ctx.resume_len != 0:
                self.report_resuming_byte(ctx.resume_len)
                request.add_header('Range', 'bytes=%d-' % ctx.resume_len)
                ctx.open_mode = 'ab'
            # Establish connection
            try:
-                data = self.ydl.urlopen(request)
+                ctx.data = self.ydl.urlopen(request)
                # When trying to resume, Content-Range HTTP header of response has to be checked
                # to match the value of requested Range HTTP header. This is due to a webservers
                # that don't support resuming and serve a whole file with no Content-Range
                # set in response despite of requested Range (see
                # https://github.com/rg3/youtube-dl/issues/6057#issuecomment-126129799)
-                if resume_len > 0:
+                if ctx.resume_len > 0:
-                    content_range = data.headers.get('Content-Range')
+                    content_range = ctx.data.headers.get('Content-Range')
                    if content_range:
                        content_range_m = re.search(r'bytes (\d+)-', content_range)
                        # Content-Range is present and matches requested Range, resume is possible
-                        if content_range_m and resume_len == int(content_range_m.group(1)):
+                        if content_range_m and ctx.resume_len == int(content_range_m.group(1)):
-                            break
+                            return
                    # Content-Range is either not present or invalid. Assuming remote webserver is
                    # trying to send the whole file, resume is not possible, so wiping the local file
                    # and performing entire redownload
                    self.report_unable_to_resume()
-                    resume_len = 0
+                    ctx.resume_len = 0
-                    open_mode = 'wb'
+                    ctx.open_mode = 'wb'
-                break
+                return
            except (compat_urllib_error.HTTPError, ) as err:
                if (err.code < 500 or err.code >= 600) and err.code != 416:
                    # Unexpected HTTP error
@ -86,15 +99,15 @@ class HttpFD(FileDownloader):
                    # Unable to resume (requested range not satisfiable)
                    try:
                        # Open the connection again without the range header
-                        data = self.ydl.urlopen(basic_request)
+                        ctx.data = self.ydl.urlopen(basic_request)
-                        content_length = data.info()['Content-Length']
+                        content_length = ctx.data.info()['Content-Length']
                    except (compat_urllib_error.HTTPError, ) as err:
                        if err.code < 500 or err.code >= 600:
                            raise
                    else:
                        # Examine the reported length
                        if (content_length is not None and
-                                (resume_len - 100 < int(content_length) < resume_len + 100)):
+                                (ctx.resume_len - 100 < int(content_length) < ctx.resume_len + 100)):
                            # The file had already been fully downloaded.
                            # Explanation to the above condition: in issue #175 it was revealed that
                            # YouTube sometimes adds or removes a few bytes from the end of the file,
@ -102,36 +115,30 @@ class HttpFD(FileDownloader):
                            # I decided to implement a suggested change and consider the file
                            # completely downloaded if the file size differs less than 100 bytes from
                            # the one in the hard drive.
-                            self.report_file_already_downloaded(filename)
+                            self.report_file_already_downloaded(ctx.filename)
-                            self.try_rename(tmpfilename, filename)
+                            self.try_rename(ctx.tmpfilename, ctx.filename)
                            self._hook_progress({
-                                'filename': filename,
+                                'filename': ctx.filename,
                                'status': 'finished',
-                                'downloaded_bytes': resume_len,
+                                'downloaded_bytes': ctx.resume_len,
-                                'total_bytes': resume_len,
+                                'total_bytes': ctx.resume_len,
                            })
-                            return True
+                            raise SucceedDownload()
                        else:
                            # The length does not match, we start the download over
                            self.report_unable_to_resume()
-                            resume_len = 0
+                            ctx.resume_len = 0
-                            open_mode = 'wb'
+                            ctx.open_mode = 'wb'
-                            break
+                            return
-            except socket.error as e:
+                raise RetryDownload(err)
-                if e.errno != errno.ECONNRESET:
+            except socket.error as err:
                if err.errno != errno.ECONNRESET:
                    # Connection reset is no problem, just retry
                    raise
                raise RetryDownload(err)
-            # Retry
+        def download():
-            count += 1
+            data_len = ctx.data.info().get('Content-length', None)
            if count <= retries:
                self.report_retry(count, retries)
        if count > retries:
            self.report_error('giving up after %s retries' % retries)
            return False
        data_len = data.info().get('Content-length', None)
            # Range HTTP header may be ignored/unsupported by a webserver
            # (e.g. extractor/scivee.py, extractor/bambuser.py).
@ -142,7 +149,7 @@ class HttpFD(FileDownloader):
                data_len = self._TEST_FILE_SIZE
            if data_len is not None:
-            data_len = int(data_len) + resume_len
+                data_len = int(data_len) + ctx.resume_len
                min_data_len = self.params.get('min_filesize')
                max_data_len = self.params.get('max_filesize')
                if min_data_len is not None and data_len < min_data_len:
@ -152,17 +159,34 @@ class HttpFD(FileDownloader):
                    self.to_screen('\r[download] File is larger than max-filesize (%s bytes > %s bytes). Aborting.' % (data_len, max_data_len))
                    return False
-        byte_counter = 0 + resume_len
+            byte_counter = 0 + ctx.resume_len
            block_size = self.params.get('buffersize', 1024)
            start = time.time()
            # measure time over whole while-loop, so slow_down() and best_block_size() work together properly
            now = None  # needed for slow_down() in the first loop run
            before = start  # start measuring
        while True:
            def retry(e):
                if ctx.tmpfilename != '-':
                    ctx.stream.close()
                ctx.stream = None
                ctx.resume_len = os.path.getsize(encodeFilename(ctx.tmpfilename))
                raise RetryDownload(e)
            while True:
                try:
                    # Download and write
-            data_block = data.read(block_size if not is_test else min(block_size, data_len - byte_counter))
+                    data_block = ctx.data.read(block_size if not is_test else min(block_size, data_len - byte_counter))
                # socket.timeout is a subclass of socket.error but may not have
                # errno set
                except socket.timeout as e:
                    retry(e)
                except socket.error as e:
                    if e.errno not in (errno.ECONNRESET, errno.ETIMEDOUT):
                        raise
                    retry(e)
                byte_counter += len(data_block)
                # exit loop when download is finished
@ -170,31 +194,32 @@ class HttpFD(FileDownloader):
                    break
                # Open destination file just in time
-            if stream is None:
+                if ctx.stream is None:
                    try:
-                    (stream, tmpfilename) = sanitize_open(tmpfilename, open_mode)
+                        ctx.stream, ctx.tmpfilename = sanitize_open(
-                    assert stream is not None
+                            ctx.tmpfilename, ctx.open_mode)
-                    filename = self.undo_temp_name(tmpfilename)
+                        assert ctx.stream is not None
-                    self.report_destination(filename)
+                        ctx.filename = self.undo_temp_name(ctx.tmpfilename)
                        self.report_destination(ctx.filename)
                    except (OSError, IOError) as err:
                        self.report_error('unable to open for writing: %s' % str(err))
                        return False
                    if self.params.get('xattr_set_filesize', False) and data_len is not None:
                        try:
-                        write_xattr(tmpfilename, 'user.ytdl.filesize', str(data_len).encode('utf-8'))
+                            write_xattr(ctx.tmpfilename, 'user.ytdl.filesize', str(data_len).encode('utf-8'))
                        except (XAttrUnavailableError, XAttrMetadataError) as err:
                            self.report_error('unable to set filesize xattr: %s' % str(err))
                try:
-                stream.write(data_block)
+                    ctx.stream.write(data_block)
                except (IOError, OSError) as err:
                    self.to_stderr('\n')
                    self.report_error('unable to write data: %s' % str(err))
                    return False
                # Apply rate limit
-            self.slow_down(start, now, byte_counter - resume_len)
+                self.slow_down(start, now, byte_counter - ctx.resume_len)
                # end measuring of one loop run
                now = time.time()
@ -207,18 +232,18 @@ class HttpFD(FileDownloader):
                before = after
                # Progress message
-            speed = self.calc_speed(start, now, byte_counter - resume_len)
+                speed = self.calc_speed(start, now, byte_counter - ctx.resume_len)
                if data_len is None:
                    eta = None
                else:
-                eta = self.calc_eta(start, time.time(), data_len - resume_len, byte_counter - resume_len)
+                    eta = self.calc_eta(start, time.time(), data_len - ctx.resume_len, byte_counter - ctx.resume_len)
                self._hook_progress({
                    'status': 'downloading',
                    'downloaded_bytes': byte_counter,
                    'total_bytes': data_len,
-                'tmpfilename': tmpfilename,
+                    'tmpfilename': ctx.tmpfilename,
-                'filename': filename,
+                    'filename': ctx.filename,
                    'eta': eta,
                    'speed': speed,
                    'elapsed': now - start,
@ -227,27 +252,47 @@ class HttpFD(FileDownloader):
                if is_test and byte_counter == data_len:
                    break
-        if stream is None:
+            if ctx.stream is None:
                self.to_stderr('\n')
                self.report_error('Did not get any data blocks')
                return False
-        if tmpfilename != '-':
+            if ctx.tmpfilename != '-':
-            stream.close()
+                ctx.stream.close()
            if data_len is not None and byte_counter != data_len:
-            raise ContentTooShortError(byte_counter, int(data_len))
+                err = ContentTooShortError(byte_counter, int(data_len))
-        self.try_rename(tmpfilename, filename)
+                if count <= retries:
                    retry(err)
                raise err
            self.try_rename(ctx.tmpfilename, ctx.filename)
            # Update file modification time
            if self.params.get('updatetime', True):
-            info_dict['filetime'] = self.try_utime(filename, data.info().get('last-modified', None))
+                info_dict['filetime'] = self.try_utime(ctx.filename, ctx.data.info().get('last-modified', None))
            self._hook_progress({
                'downloaded_bytes': byte_counter,
                'total_bytes': byte_counter,
-            'filename': filename,
+                'filename': ctx.filename,
                'status': 'finished',
                'elapsed': time.time() - start,
            })
            return True
        while count <= retries:
            try:
                establish_connection()
                download()
                return True
            except RetryDownload as e:
                count += 1
                if count <= retries:
                    self.report_retry(e.source_error, count, retries)
                continue
            except SucceedDownload:
                return True
        self.report_error('giving up after %s retries' % retries)
        return False
--- a/youtube_dl/extractor/bandcamp.py
+++ b/youtube_dl/extractor/bandcamp.py
@ -242,7 +242,12 @@ class BandcampAlbumIE(InfoExtractor):
            raise ExtractorError('The page doesn\'t contain any tracks')
        # Only tracks with duration info have songs
        entries = [
-            self.url_result(compat_urlparse.urljoin(url, t_path), ie=BandcampIE.ie_key())
+            self.url_result(
                compat_urlparse.urljoin(url, t_path),
                ie=BandcampIE.ie_key(),
                video_title=self._search_regex(
                    r'<span\b[^>]+\bitemprop=["\']name["\'][^>]*>([^<]+)',
                    elem_content, 'track title', fatal=False))
            for elem_content, t_path in track_elements
            if self._html_search_meta('duration', elem_content, default=None)]
--- a/youtube_dl/extractor/bbc.py
+++ b/youtube_dl/extractor/bbc.py
@ -29,7 +29,7 @@ from ..compat import (
 class BBCCoUkIE(InfoExtractor):
    IE_NAME = 'bbc.co.uk'
    IE_DESC = 'BBC iPlayer'
-    _ID_REGEX = r'[pb][\da-z]{7}'
+    _ID_REGEX = r'[pbw][\da-z]{7}'
    _VALID_URL = r'''(?x)
                    https?://
                        (?:www\.)?bbc\.co\.uk/
@ -37,7 +37,8 @@ class BBCCoUkIE(InfoExtractor):
                            programmes/(?!articles/)|
                            iplayer(?:/[^/]+)?/(?:episode/|playlist/)|
                            music/(?:clips|audiovideo/popular)[/#]|
-                            radio/player/
+                            radio/player/|
                            events/[^/]+/play/[^/]+/
                        )
                        (?P<id>%s)(?!/(?:episodes|broadcasts|clips))
                    ''' % _ID_REGEX
@ -232,6 +233,9 @@ class BBCCoUkIE(InfoExtractor):
        }, {
            'url': 'https://www.bbc.co.uk/music/audiovideo/popular#p055bc55',
            'only_matching': True,
        }, {
            'url': 'http://www.bbc.co.uk/programmes/w3csv1y9',
            'only_matching': True,
        }]
    _USP_RE = r'/([^/]+?)\.ism(?:\.hlsv2\.ism)?/[^/]+\.m3u8'
--- a/youtube_dl/extractor/cbc.py
+++ b/youtube_dl/extractor/cbc.py
@ -200,6 +200,7 @@ class CBCWatchBaseIE(InfoExtractor):
        'media': 'http://search.yahoo.com/mrss/',
        'clearleap': 'http://www.clearleap.com/namespace/clearleap/1.0/',
    }
    _GEO_COUNTRIES = ['CA']
    def _call_api(self, path, video_id):
        url = path if path.startswith('http') else self._API_BASE_URL + path
@ -287,6 +288,11 @@ class CBCWatchBaseIE(InfoExtractor):
 class CBCWatchVideoIE(CBCWatchBaseIE):
    IE_NAME = 'cbc.ca:watch:video'
    _VALID_URL = r'https?://api-cbc\.cloud\.clearleap\.com/cloffice/client/web/play/?\?.*?\bcontentId=(?P<id>[\da-f]{8}-[\da-f]{4}-[\da-f]{4}-[\da-f]{4}-[\da-f]{12})'
    _TEST = {
        # geo-restricted to Canada, bypassable
        'url': 'https://api-cbc.cloud.clearleap.com/cloffice/client/web/play/?contentId=3c84472a-1eea-4dee-9267-2655d5055dcf&categoryId=ebc258f5-ee40-4cca-b66b-ba6bd55b7235',
        'only_matching': True,
    }
    def _real_extract(self, url):
        video_id = self._match_id(url)
@ -323,9 +329,10 @@ class CBCWatchIE(CBCWatchBaseIE):
    IE_NAME = 'cbc.ca:watch'
    _VALID_URL = r'https?://watch\.cbc\.ca/(?:[^/]+/)+(?P<id>[0-9a-f-]+)'
    _TESTS = [{
        # geo-restricted to Canada, bypassable
        'url': 'http://watch.cbc.ca/doc-zone/season-6/customer-disservice/38e815a-009e3ab12e4',
        'info_dict': {
-            'id': '38e815a-009e3ab12e4',
+            'id': '9673749a-5e77-484c-8b62-a1092a6b5168',
            'ext': 'mp4',
            'title': 'Customer (Dis)Service',
            'description': 'md5:8bdd6913a0fe03d4b2a17ebe169c7c87',
@ -337,8 +344,8 @@ class CBCWatchIE(CBCWatchBaseIE):
            'skip_download': True,
            'format': 'bestvideo',
        },
        'skip': 'Geo-restricted to Canada',
    }, {
        # geo-restricted to Canada, bypassable
        'url': 'http://watch.cbc.ca/arthur/all/1ed4b385-cd84-49cf-95f0-80f004680057',
        'info_dict': {
            'id': '1ed4b385-cd84-49cf-95f0-80f004680057',
@ -346,7 +353,6 @@ class CBCWatchIE(CBCWatchBaseIE):
            'description': 'Arthur, the sweetest 8-year-old aardvark, and his pals solve all kinds of problems with humour, kindness and teamwork.',
        },
        'playlist_mincount': 30,
        'skip': 'Geo-restricted to Canada',
    }]
    def _real_extract(self, url):
--- a/youtube_dl/extractor/cda.py
+++ b/youtube_dl/extractor/cda.py
@ -124,7 +124,7 @@ class CDAIE(InfoExtractor):
        }
        def extract_format(page, version):
-            json_str = self._search_regex(
+            json_str = self._html_search_regex(
                r'player_data=(\\?["\'])(?P<player_data>.+?)\1', page,
                '%s player_json' % version, fatal=False, group='player_data')
            if not json_str:
--- a/youtube_dl/extractor/charlierose.py
+++ b/youtube_dl/extractor/charlierose.py
@ -5,7 +5,7 @@ from ..utils import remove_end
 class CharlieRoseIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?charlierose\.com/video(?:s|/player)/(?P<id>\d+)'
+    _VALID_URL = r'https?://(?:www\.)?charlierose\.com/(?:video|episode)(?:s|/player)/(?P<id>\d+)'
    _TESTS = [{
        'url': 'https://charlierose.com/videos/27996',
        'md5': 'fda41d49e67d4ce7c2411fd2c4702e09',
@ -24,6 +24,9 @@ class CharlieRoseIE(InfoExtractor):
    }, {
        'url': 'https://charlierose.com/videos/27996',
        'only_matching': True,
    }, {
        'url': 'https://charlierose.com/episodes/30887?autoplay=true',
        'only_matching': True,
    }]
    _PLAYER_BASE = 'https://charlierose.com/video/player/%s'
--- a/youtube_dl/extractor/common.py
+++ b/youtube_dl/extractor/common.py
@ -27,6 +27,7 @@ from ..compat import (
    compat_urllib_parse_urlencode,
    compat_urllib_request,
    compat_urlparse,
    compat_xml_parse_error,
 )
 from ..downloader.f4m import remove_encrypted_media
 from ..utils import (
@ -646,15 +647,29 @@ class InfoExtractor(object):
    def _download_xml(self, url_or_request, video_id,
                      note='Downloading XML', errnote='Unable to download XML',
-                      transform_source=None, fatal=True, encoding=None, data=None, headers={}, query={}):
+                      transform_source=None, fatal=True, encoding=None,
                      data=None, headers={}, query={}):
        """Return the xml as an xml.etree.ElementTree.Element"""
        xml_string = self._download_webpage(
-            url_or_request, video_id, note, errnote, fatal=fatal, encoding=encoding, data=data, headers=headers, query=query)
+            url_or_request, video_id, note, errnote, fatal=fatal,
            encoding=encoding, data=data, headers=headers, query=query)
        if xml_string is False:
            return xml_string
        return self._parse_xml(
            xml_string, video_id, transform_source=transform_source,
            fatal=fatal)
    def _parse_xml(self, xml_string, video_id, transform_source=None, fatal=True):
        if transform_source:
            xml_string = transform_source(xml_string)
        try:
            return compat_etree_fromstring(xml_string.encode('utf-8'))
        except compat_xml_parse_error as ve:
            errmsg = '%s: Failed to parse XML ' % video_id
            if fatal:
                raise ExtractorError(errmsg, cause=ve)
            else:
                self.report_warning(errmsg + str(ve))
    def _download_json(self, url_or_request, video_id,
                       note='Downloading JSON metadata',
@ -2123,11 +2138,11 @@ class InfoExtractor(object):
                formats = self._extract_m3u8_formats(
                    full_url, video_id, ext='mp4',
                    entry_protocol=m3u8_entry_protocol, m3u8_id=m3u8_id,
-                    preference=preference)
+                    preference=preference, fatal=False)
            elif ext == 'mpd':
                is_plain_url = False
                formats = self._extract_mpd_formats(
-                    full_url, video_id, mpd_id=mpd_id)
+                    full_url, video_id, mpd_id=mpd_id, fatal=False)
            else:
                is_plain_url = True
                formats = [{
@ -2169,6 +2184,12 @@ class InfoExtractor(object):
                    f = parse_content_type(source_attributes.get('type'))
                    is_plain_url, formats = _media_formats(src, media_type, f)
                    if is_plain_url:
                        # res attribute is not standard but seen several times
                        # in the wild
                        f.update({
                            'height': int_or_none(source_attributes.get('res')),
                            'format_id': source_attributes.get('label'),
                        })
                        f.update(formats[0])
                        media_info['formats'].append(f)
                    else:
--- a/youtube_dl/extractor/extractors.py
+++ b/youtube_dl/extractor/extractors.py
@ -509,6 +509,7 @@ from .la7 import LA7IE
 from .laola1tv import (
    Laola1TvEmbedIE,
    Laola1TvIE,
    ITTFIE,
 )
 from .lci import LCIIE
 from .lcp import (
@ -536,7 +537,10 @@ from .limelight import (
    LimelightChannelListIE,
 )
 from .litv import LiTVIE
-from .liveleak import LiveLeakIE
+from .liveleak import (
    LiveLeakIE,
    LiveLeakEmbedIE,
 )
 from .livestream import (
    LivestreamIE,
    LivestreamOriginalIE,
--- a/youtube_dl/extractor/generic.py
+++ b/youtube_dl/extractor/generic.py
@ -1519,14 +1519,27 @@ class GenericIE(InfoExtractor):
        # LiveLeak embed
        {
            'url': 'http://www.wykop.pl/link/3088787/',
-            'md5': 'ace83b9ed19b21f68e1b50e844fdf95d',
+            'md5': '7619da8c820e835bef21a1efa2a0fc71',
            'info_dict': {
                'id': '874_1459135191',
                'ext': 'mp4',
                'title': 'Man shows poor quality of new apartment building',
                'description': 'The wall is like a sand pile.',
                'uploader': 'Lake8737',
-            }
+            },
            'add_ie': [LiveLeakIE.ie_key()],
        },
        # Another LiveLeak embed pattern (#13336)
        {
            'url': 'https://milo.yiannopoulos.net/2017/06/concealed-carry-robbery/',
            'info_dict': {
                'id': '2eb_1496309988',
                'ext': 'mp4',
                'title': 'Thief robs place where everyone was armed',
                'description': 'md5:694d73ee79e535953cf2488562288eee',
                'uploader': 'brazilwtf',
            },
            'add_ie': [LiveLeakIE.ie_key()],
        },
        # Duplicated embedded video URLs
        {
@ -2757,9 +2770,9 @@ class GenericIE(InfoExtractor):
                self._proto_relative_url(instagram_embed_url), InstagramIE.ie_key())
        # Look for LiveLeak embeds
-        liveleak_url = LiveLeakIE._extract_url(webpage)
+        liveleak_urls = LiveLeakIE._extract_urls(webpage)
-        if liveleak_url:
+        if liveleak_urls:
-            return self.url_result(liveleak_url, 'LiveLeak')
+            return self.playlist_from_matches(liveleak_urls, video_id, video_title)
        # Look for 3Q SDN embeds
        threeqsdn_url = ThreeQSDNIE._extract_url(webpage)
@ -2858,12 +2871,6 @@ class GenericIE(InfoExtractor):
                    merged[k] = v
            return merged
        # Looking for http://schema.org/VideoObject
        json_ld = self._search_json_ld(
            webpage, video_id, default={}, expected_type='VideoObject')
        if json_ld.get('url'):
            return merge_dicts(json_ld, info_dict)
        # Look for HTML5 media
        entries = self._parse_html5_media_entries(url, webpage, video_id, m3u8_id='hls')
        if entries:
@ -2882,6 +2889,12 @@ class GenericIE(InfoExtractor):
                jwplayer_data, video_id, require_title=False, base_url=url)
            return merge_dicts(info, info_dict)
        # Looking for http://schema.org/VideoObject
        json_ld = self._search_json_ld(
            webpage, video_id, default={}, expected_type='VideoObject')
        if json_ld.get('url'):
            return merge_dicts(json_ld, info_dict)
        def check_video(vurl):
            if YoutubeIE.suitable(vurl):
                return True
--- a/youtube_dl/extractor/googledrive.py
+++ b/youtube_dl/extractor/googledrive.py
@ -4,26 +4,61 @@ import re
 from .common import InfoExtractor
 from ..utils import (
    determine_ext,
    ExtractorError,
    int_or_none,
    lowercase_escape,
    update_url_query,
 )
 class GoogleDriveIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:(?:docs|drive)\.google\.com/(?:uc\?.*?id=|file/d/)|video\.google\.com/get_player\?.*?docid=)(?P<id>[a-zA-Z0-9_-]{28,})'
+    _VALID_URL = r'''(?x)
                        https?://
                            (?:
                                (?:docs|drive)\.google\.com/
                                (?:
                                    (?:uc|open)\?.*?id=|
                                    file/d/
                                )|
                                video\.google\.com/get_player\?.*?docid=
                            )
                            (?P<id>[a-zA-Z0-9_-]{28,})
                    '''
    _TESTS = [{
        'url': 'https://drive.google.com/file/d/0ByeS4oOUV-49Zzh4R1J6R09zazQ/edit?pli=1',
-        'md5': 'd109872761f7e7ecf353fa108c0dbe1e',
+        'md5': '5c602afbbf2c1db91831f5d82f678554',
        'info_dict': {
            'id': '0ByeS4oOUV-49Zzh4R1J6R09zazQ',
            'ext': 'mp4',
            'title': 'Big Buck Bunny.mp4',
            'duration': 45,
        }
    }, {
        # video can't be watched anonymously due to view count limit reached,
        # but can be downloaded (see https://github.com/rg3/youtube-dl/issues/14046)
        'url': 'https://drive.google.com/file/d/0B-vUyvmDLdWDcEt4WjBqcmI2XzQ/view',
        'md5': 'bfbd670d03a470bb1e6d4a257adec12e',
        'info_dict': {
            'id': '0B-vUyvmDLdWDcEt4WjBqcmI2XzQ',
            'ext': 'mp4',
            'title': 'Annabelle Creation (2017)- Z.V1 [TH].MP4',
        }
    }, {
        # video id is longer than 28 characters
        'url': 'https://drive.google.com/file/d/1ENcQ_jeCuj7y19s66_Ou9dRP4GKGsodiDQ/edit',
        'info_dict': {
            'id': '1ENcQ_jeCuj7y19s66_Ou9dRP4GKGsodiDQ',
            'ext': 'mp4',
            'title': 'Andreea Banica feat Smiley - Hooky Song (Official Video).mp4',
            'duration': 189,
        },
        'only_matching': True,
    }, {
        'url': 'https://drive.google.com/open?id=0B2fjwgkl1A_CX083Tkowdmt6d28',
        'only_matching': True,
    }, {
        'url': 'https://drive.google.com/uc?id=0B2fjwgkl1A_CX083Tkowdmt6d28',
        'only_matching': True,
    }]
    _FORMATS_EXT = {
@ -44,6 +79,13 @@ class GoogleDriveIE(InfoExtractor):
        '46': 'webm',
        '59': 'mp4',
    }
    _BASE_URL_CAPTIONS = 'https://drive.google.com/timedtext'
    _CAPTIONS_ENTRY_TAG = {
        'subtitles': 'track',
        'automatic_captions': 'target',
    }
    _caption_formats_ext = []
    _captions_xml = None
    @staticmethod
    def _extract_url(webpage):
@ -53,22 +95,100 @@ class GoogleDriveIE(InfoExtractor):
        if mobj:
            return 'https://drive.google.com/file/d/%s' % mobj.group('id')
    def _download_subtitles_xml(self, video_id, subtitles_id, hl):
        if self._captions_xml:
            return
        self._captions_xml = self._download_xml(
            self._BASE_URL_CAPTIONS, video_id, query={
                'id': video_id,
                'vid': subtitles_id,
                'hl': hl,
                'v': video_id,
                'type': 'list',
                'tlangs': '1',
                'fmts': '1',
                'vssids': '1',
            }, note='Downloading subtitles XML',
            errnote='Unable to download subtitles XML', fatal=False)
        if self._captions_xml:
            for f in self._captions_xml.findall('format'):
                if f.attrib.get('fmt_code') and not f.attrib.get('default'):
                    self._caption_formats_ext.append(f.attrib['fmt_code'])
    def _get_captions_by_type(self, video_id, subtitles_id, caption_type,
                              origin_lang_code=None):
        if not subtitles_id or not caption_type:
            return
        captions = {}
        for caption_entry in self._captions_xml.findall(
                self._CAPTIONS_ENTRY_TAG[caption_type]):
            caption_lang_code = caption_entry.attrib.get('lang_code')
            if not caption_lang_code:
                continue
            caption_format_data = []
            for caption_format in self._caption_formats_ext:
                query = {
                    'vid': subtitles_id,
                    'v': video_id,
                    'fmt': caption_format,
                    'lang': (caption_lang_code if origin_lang_code is None
                             else origin_lang_code),
                    'type': 'track',
                    'name': '',
                    'kind': '',
                }
                if origin_lang_code is not None:
                    query.update({'tlang': caption_lang_code})
                caption_format_data.append({
                    'url': update_url_query(self._BASE_URL_CAPTIONS, query),
                    'ext': caption_format,
                })
            captions[caption_lang_code] = caption_format_data
        return captions
    def _get_subtitles(self, video_id, subtitles_id, hl):
        if not subtitles_id or not hl:
            return
        self._download_subtitles_xml(video_id, subtitles_id, hl)
        if not self._captions_xml:
            return
        return self._get_captions_by_type(video_id, subtitles_id, 'subtitles')
    def _get_automatic_captions(self, video_id, subtitles_id, hl):
        if not subtitles_id or not hl:
            return
        self._download_subtitles_xml(video_id, subtitles_id, hl)
        if not self._captions_xml:
            return
        track = self._captions_xml.find('track')
        if track is None:
            return
        origin_lang_code = track.attrib.get('lang_code')
        if not origin_lang_code:
            return
        return self._get_captions_by_type(
            video_id, subtitles_id, 'automatic_captions', origin_lang_code)
    def _real_extract(self, url):
        video_id = self._match_id(url)
        webpage = self._download_webpage(
            'http://docs.google.com/file/d/%s' % video_id, video_id)
-        reason = self._search_regex(r'"reason"\s*,\s*"([^"]+)', webpage, 'reason', default=None)
+        title = self._search_regex(
-        if reason:
+            r'"title"\s*,\s*"([^"]+)', webpage, 'title',
-            raise ExtractorError(reason)
+            default=None) or self._og_search_title(webpage)
        title = self._search_regex(r'"title"\s*,\s*"([^"]+)', webpage, 'title')
        duration = int_or_none(self._search_regex(
-            r'"length_seconds"\s*,\s*"([^"]+)', webpage, 'length seconds', default=None))
+            r'"length_seconds"\s*,\s*"([^"]+)', webpage, 'length seconds',
-        fmt_stream_map = self._search_regex(
+            default=None))
            r'"fmt_stream_map"\s*,\s*"([^"]+)', webpage, 'fmt stream map').split(',')
        fmt_list = self._search_regex(r'"fmt_list"\s*,\s*"([^"]+)', webpage, 'fmt_list').split(',')
        formats = []
        fmt_stream_map = self._search_regex(
            r'"fmt_stream_map"\s*,\s*"([^"]+)', webpage,
            'fmt stream map', default='').split(',')
        fmt_list = self._search_regex(
            r'"fmt_list"\s*,\s*"([^"]+)', webpage,
            'fmt_list', default='').split(',')
        if fmt_stream_map and fmt_list:
            resolutions = {}
            for fmt in fmt_list:
                mobj = re.search(
@ -77,7 +197,6 @@ class GoogleDriveIE(InfoExtractor):
                    resolutions[mobj.group('format_id')] = (
                        int(mobj.group('width')), int(mobj.group('height')))
        formats = []
            for fmt_stream in fmt_stream_map:
                fmt_stream_split = fmt_stream.split('|')
                if len(fmt_stream_split) < 2:
@ -95,12 +214,64 @@ class GoogleDriveIE(InfoExtractor):
                        'height': resolution[1],
                    })
                formats.append(f)
        source_url = update_url_query(
            'https://drive.google.com/uc', {
                'id': video_id,
                'export': 'download',
            })
        urlh = self._request_webpage(
            source_url, video_id, note='Requesting source file',
            errnote='Unable to request source file', fatal=False)
        if urlh:
            def add_source_format(src_url):
                formats.append({
                    'url': src_url,
                    'ext': determine_ext(title, 'mp4').lower(),
                    'format_id': 'source',
                    'quality': 1,
                })
            if urlh.headers.get('Content-Disposition'):
                add_source_format(source_url)
            else:
                confirmation_webpage = self._webpage_read_content(
                    urlh, url, video_id, note='Downloading confirmation page',
                    errnote='Unable to confirm download', fatal=False)
                if confirmation_webpage:
                    confirm = self._search_regex(
                        r'confirm=([^&"\']+)', confirmation_webpage,
                        'confirmation code', fatal=False)
                    if confirm:
                        add_source_format(update_url_query(source_url, {
                            'confirm': confirm,
                        }))
        if not formats:
            reason = self._search_regex(
                r'"reason"\s*,\s*"([^"]+)', webpage, 'reason', default=None)
            if reason:
                raise ExtractorError(reason, expected=True)
        self._sort_formats(formats)
        hl = self._search_regex(
            r'"hl"\s*,\s*"([^"]+)', webpage, 'hl', default=None)
        subtitles_id = None
        ttsurl = self._search_regex(
            r'"ttsurl"\s*,\s*"([^"]+)', webpage, 'ttsurl', default=None)
        if ttsurl:
            # the video Id for subtitles will be the last value in the ttsurl
            # query string
            subtitles_id = ttsurl.encode('utf-8').decode(
                'unicode_escape').split('=')[-1]
        return {
            'id': video_id,
            'title': title,
            'thumbnail': self._og_search_thumbnail(webpage, default=None),
            'duration': duration,
            'formats': formats,
            'subtitles': self.extract_subtitles(video_id, subtitles_id, hl),
            'automatic_captions': self.extract_automatic_captions(
                video_id, subtitles_id, hl),
        }
--- a/youtube_dl/extractor/laola1tv.py
+++ b/youtube_dl/extractor/laola1tv.py
@ -215,3 +215,21 @@ class Laola1TvIE(Laola1TvEmbedIE):
            'formats': formats,
            'is_live': is_live,
        }
 class ITTFIE(InfoExtractor):
    _VALID_URL = r'https?://tv\.ittf\.com/video/[^/]+/(?P<id>\d+)'
    _TEST = {
        'url': 'https://tv.ittf.com/video/peng-wang-wei-matsudaira-kenta/951802',
        'only_matching': True,
    }
    def _real_extract(self, url):
        return self.url_result(
            update_url_query('https://www.laola1.tv/titanplayer.php', {
                'videoid': self._match_id(url),
                'type': 'V',
                'lang': 'en',
                'portal': 'int',
                'customer': 1024,
            }), Laola1TvEmbedIE.ie_key())
--- a/youtube_dl/extractor/liveleak.py
+++ b/youtube_dl/extractor/liveleak.py
@ -72,15 +72,20 @@ class LiveLeakIE(InfoExtractor):
        'params': {
            'skip_download': True,
        },
    }, {
        'url': 'https://www.liveleak.com/view?i=677_1439397581',
        'info_dict': {
            'id': '677_1439397581',
            'title': 'Fuel Depot in China Explosion caught on video',
        },
        'playlist_count': 3,
    }]
    @staticmethod
-    def _extract_url(webpage):
+    def _extract_urls(webpage):
-        mobj = re.search(
+        return re.findall(
-            r'<iframe[^>]+src="https?://(?:\w+\.)?liveleak\.com/ll_embed\?(?:.*?)i=(?P<id>[\w_]+)(?:.*)',
+            r'<iframe[^>]+src="(https?://(?:\w+\.)?liveleak\.com/ll_embed\?[^"]*[if]=[\w_]+[^"]+)"',
            webpage)
        if mobj:
            return 'http://www.liveleak.com/view?i=%s' % mobj.group('id')
    def _real_extract(self, url):
        video_id = self._match_id(url)
@ -111,8 +116,7 @@ class LiveLeakIE(InfoExtractor):
                'age_limit': age_limit,
            }
-        info_dict = entries[0]
+        for idx, info_dict in enumerate(entries):
            for a_format in info_dict['formats']:
                if not a_format.get('height'):
                    a_format['height'] = int_or_none(self._search_regex(
@ -121,8 +125,13 @@ class LiveLeakIE(InfoExtractor):
            self._sort_formats(info_dict['formats'])
            # Don't append entry ID for one-video pages to keep backward compatibility
            if len(entries) > 1:
                info_dict['id'] = '%s_%s' % (video_id, idx + 1)
            else:
                info_dict['id'] = video_id
            info_dict.update({
            'id': video_id,
                'title': video_title,
                'description': video_description,
                'uploader': video_uploader,
@ -130,4 +139,31 @@ class LiveLeakIE(InfoExtractor):
                'thumbnail': video_thumbnail,
            })
-        return info_dict
+        return self.playlist_result(entries, video_id, video_title)
 class LiveLeakEmbedIE(InfoExtractor):
    _VALID_URL = r'https?://(?:www\.)?liveleak\.com/ll_embed\?.*?\b(?P<kind>[if])=(?P<id>[\w_]+)'
    # See generic.py for actual test cases
    _TESTS = [{
        'url': 'https://www.liveleak.com/ll_embed?i=874_1459135191',
        'only_matching': True,
    }, {
        'url': 'https://www.liveleak.com/ll_embed?f=ab065df993c1',
        'only_matching': True,
    }]
    def _real_extract(self, url):
        mobj = re.match(self._VALID_URL, url)
        kind, video_id = mobj.group('kind', 'id')
        if kind == 'f':
            webpage = self._download_webpage(url, video_id)
            liveleak_url = self._search_regex(
                r'logourl\s*:\s*(?P<q1>[\'"])(?P<url>%s)(?P=q1)' % LiveLeakIE._VALID_URL,
                webpage, 'LiveLeak URL', group='url')
        elif kind == 'i':
            liveleak_url = 'http://www.liveleak.com/view?i=%s' % video_id
        return self.url_result(liveleak_url, ie=LiveLeakIE.ie_key())
--- a/youtube_dl/extractor/mixcloud.py
+++ b/youtube_dl/extractor/mixcloud.py
@ -92,7 +92,7 @@ class MixcloudIE(InfoExtractor):
                js = self._download_webpage(js_url, track_id, fatal=False)
                if js:
                    KEY_RE_TEMPLATE = r'player\s*:\s*{.*?\b%s\s*:\s*(["\'])(?P<key>(?:(?!\1).)+)\1'
-                    for key_name in ('value', 'key_value'):
+                    for key_name in ('value', 'key_value', 'key_value.*?', '.*?value.*?'):
                        key = self._search_regex(
                            KEY_RE_TEMPLATE % key_name, js, 'key',
                            default=None, group='key')
--- a/youtube_dl/extractor/pornhd.py
+++ b/youtube_dl/extractor/pornhd.py
@ -54,7 +54,7 @@ class PornHdIE(InfoExtractor):
             r'<title>(.+?) - .*?[Pp]ornHD.*?</title>'], webpage, 'title')
        sources = self._parse_json(js_to_json(self._search_regex(
-            r"(?s)sources'?\s*:\s*(\{.+?\})\s*\}[;,)]",
+            r"(?s)sources'?\s*[:=]\s*(\{.+?\})",
            webpage, 'sources', default='{}')), video_id)
        if not sources:
@ -82,7 +82,8 @@ class PornHdIE(InfoExtractor):
        view_count = int_or_none(self._html_search_regex(
            r'(\d+) views\s*<', webpage, 'view count', fatal=False))
        thumbnail = self._search_regex(
-            r"'poster'\s*:\s*'([^']+)'", webpage, 'thumbnail', fatal=False)
+            r"poster'?\s*:\s*([\"'])(?P<url>(?:(?!\1).)+)\1", webpage,
            'thumbnail', fatal=False, group='url')
        return {
            'id': video_id,
--- a/youtube_dl/extractor/pornhub.py
+++ b/youtube_dl/extractor/pornhub.py
@ -186,7 +186,7 @@ class PornHubIE(InfoExtractor):
            title, thumbnail, duration = [None] * 3
        video_uploader = self._html_search_regex(
-            r'(?s)From:&nbsp;.+?<(?:a href="/users/|a href="/channels/|span class="username)[^>]+>(.+?)<',
+            r'(?s)From:&nbsp;.+?<(?:a\b[^>]+\bhref=["\']/(?:user|channel)s/|span\b[^>]+\bclass=["\']username)[^>]+>(.+?)<',
            webpage, 'uploader', fatal=False)
        view_count = self._extract_count(
--- a/youtube_dl/extractor/rai.py
+++ b/youtube_dl/extractor/rai.py
@ -345,11 +345,11 @@ class RaiIE(RaiBaseIE):
        media_type = media['type']
        if 'Audio' in media_type:
            relinker_info = {
-                'formats': {
+                'formats': [{
                    'format_id': media.get('formatoAudio'),
                    'url': media['audioUrl'],
                    'ext': media.get('formatoAudio'),
-                }
+                }]
            }
        elif 'Video' in media_type:
            relinker_info = self._extract_relinker_info(media['mediaUri'], content_id)
--- a/youtube_dl/extractor/toutv.py
+++ b/youtube_dl/extractor/toutv.py
@ -5,7 +5,6 @@ from .common import InfoExtractor
 from ..utils import (
    int_or_none,
    js_to_json,
    ExtractorError,
    urlencode_postdata,
    extract_attributes,
    smuggle_url,
@ -78,8 +77,10 @@ class TouTvIE(InfoExtractor):
    def _real_extract(self, url):
        path = self._match_id(url)
        metadata = self._download_json('http://ici.tou.tv/presentation/%s' % path, path)
        # IsDrm does not necessarily mean the video is DRM protected (see
        # https://github.com/rg3/youtube-dl/issues/13994).
        if metadata.get('IsDrm'):
-            raise ExtractorError('This video is DRM protected.', expected=True)
+            self.report_warning('This video is probably DRM protected.', path)
        video_id = metadata['IdMedia']
        details = metadata['Details']
        title = details['OriginalTitle']
--- a/youtube_dl/extractor/youtube.py
+++ b/youtube_dl/extractor/youtube.py
@ -1003,6 +1003,27 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                'Skipping DASH manifest',
            ],
        },
        {
            # The following content has been identified by the YouTube community
            # as inappropriate or offensive to some audiences.
            'url': 'https://www.youtube.com/watch?v=6SJNVb0GnPI',
            'info_dict': {
                'id': '6SJNVb0GnPI',
                'ext': 'mp4',
                'title': 'Race Differences in Intelligence',
                'description': 'md5:5d161533167390427a1f8ee89a1fc6f1',
                'duration': 965,
                'upload_date': '20140124',
                'uploader': 'New Century Foundation',
                'uploader_id': 'UCEJYpZGqgUob0zVVEaLhvVg',
                'uploader_url': r're:https?://(?:www\.)?youtube\.com/channel/UCEJYpZGqgUob0zVVEaLhvVg',
                'license': 'Standard YouTube License',
                'view_count': int,
            },
            'params': {
                'skip_download': True,
            },
        },
        {
            # itag 212
            'url': '1t24XAntNCY',
@ -1437,9 +1458,14 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
            if dash_mpd and dash_mpd[0] not in dash_mpds:
                dash_mpds.append(dash_mpd[0])
        is_live = None
        view_count = None
        def extract_view_count(v_info):
            return int_or_none(try_get(v_info, lambda x: x['view_count'][0]))
        # Get video info
        embed_webpage = None
        is_live = None
        if re.search(r'player-age-gate-content">', video_webpage) is not None:
            age_gate = True
            # We simulate the access to the video from www.youtube.com/v/{video_id}
@ -1509,6 +1535,8 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                        continue
                    get_video_info = compat_parse_qs(video_info_webpage)
                    add_dash_mpd(get_video_info)
                    if view_count is None:
                        view_count = extract_view_count(get_video_info)
                    if not video_info:
                        video_info = get_video_info
                    if 'token' in get_video_info:
@ -1592,10 +1620,8 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
                return self.playlist_result(entries, video_id, video_title, video_description)
            self.to_screen('Downloading just video %s because of --no-playlist' % video_id)
-        if 'view_count' in video_info:
+        if view_count is None:
-            view_count = int(video_info['view_count'][0])
+            view_count = extract_view_count(video_info)
        else:
            view_count = None
        # Check for "rental" videos
        if 'ypc_video_rental_bar_text' in video_info and 'author' not in video_info:
@ -1639,10 +1665,8 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
        if not upload_date:
            upload_date = self._search_regex(
                [r'(?s)id="eow-date.*?>(.*?)</span>',
-                 r'id="watch-uploader-info".*?>.*?(?:Published|Uploaded|Streamed live|Started) on (.+?)</strong>'],
+                 r'(?:id="watch-uploader-info".*?>.*?|["\']simpleText["\']\s*:\s*["\'])(?:Published|Uploaded|Streamed live|Started) on (.+?)[<"\']'],
                video_webpage, 'upload date', default=None)
            if upload_date:
                upload_date = ' '.join(re.sub(r'[/,-]', r' ', mobj.group(1)).split())
        upload_date = unified_strdate(upload_date)
        video_license = self._html_search_regex(
--- a/youtube_dl/utils.py
+++ b/youtube_dl/utils.py
@ -596,7 +596,7 @@ def unescapeHTML(s):
    assert type(s) == compat_str
    return re.sub(
-        r'&([^;]+;)', lambda m: _htmlentity_transform(m.group(1)), s)
+        r'&([^&;]+;)', lambda m: _htmlentity_transform(m.group(1)), s)
 def get_subprocess_encoding():
--- a/youtube_dl/version.py
+++ b/youtube_dl/version.py
@ -1,3 +1,3 @@
 from __future__ import unicode_literals
-__version__ = '2017.08.18'
+__version__ = '2017.08.27.1'
`@ -1,3 +1,3 @@`
	`from __future__ import unicode_literals`	`from __future__ import unicode_literals`

	`__version__ = '2017.08.18'`	`__version__ = '2017.08.27.1'`