Merge remote-tracking branch 'upstream/master'

2025-02-03 06:56:05 +08:00 · 2018-10-07 00:21:01 +05:30 · 2018-10-07 00:21:01 +05:30 · 6fc8bdfb57
commit 6fc8bdfb57
parent de60d4966c 19a352854f
243 changed files with 10425 additions and 4256 deletions
--- a/.github/ISSUE_TEMPLATE.md
+++ b/.github/ISSUE_TEMPLATE.md
@ -6,8 +6,8 @@
 ---
-### Make sure you are using the *latest* version: run `youtube-dl --version` and ensure your version is *2018.03.20*. If it's not, read [this FAQ entry](https://github.com/rg3/youtube-dl/blob/master/README.md#how-do-i-update-youtube-dl) and update. Issues with outdated version will be rejected.
+### Make sure you are using the *latest* version: run `youtube-dl --version` and ensure your version is *2018.10.05*. If it's not, read [this FAQ entry](https://github.com/rg3/youtube-dl/blob/master/README.md#how-do-i-update-youtube-dl) and update. Issues with outdated version will be rejected.
- [ ] I've **verified** and **I assure** that I'm running youtube-dl **2018.03.20**
+- [ ] I've **verified** and **I assure** that I'm running youtube-dl **2018.10.05**
 ### Before submitting an *issue* make sure you have:
 - [ ] At least skimmed through the [README](https://github.com/rg3/youtube-dl/blob/master/README.md), **most notably** the [FAQ](https://github.com/rg3/youtube-dl#faq) and [BUGS](https://github.com/rg3/youtube-dl#bugs) sections
@ -36,7 +36,7 @@ Add the `-v` flag to **your command line** you run youtube-dl with (`youtube-dl
 [debug] User config: []
 [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
 [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
-[debug] youtube-dl version 2018.03.20
+[debug] youtube-dl version 2018.10.05
 [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
 [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
 [debug] Proxy map: {}
--- a/.gitignore
+++ b/.gitignore
@ -47,3 +47,7 @@ youtube-dl.zsh
 *.iml
 tmp/
 venv/
 # VS Code related files
 .vscode
--- a/10
+++ b/10
@ -236,3 +236,13 @@ Lei Wang
 Petr Novák
 Leonardo Taccari
 Martin Weinelt
 Surya Oktafendri
 TingPing
 Alexandre Macabies
 Bastian de Groot
 Niklas Haas
 András Veres-Szentkirályi
 Enes Solak
 Nathan Rossi
 Thomas van der Berg
 Luca Cherubin
--- a/552
+++ b/552
@ -1,3 +1,555 @@
 version 2018.10.05
 Extractors
 * [pluralsight] Improve authentication (#17762)
 * [dailymotion] Fix extraction (#17699)
 * [crunchyroll] Switch to HTTPS for RpcApi (#17749)
 + [philharmoniedeparis] Add support for pad.philharmoniedeparis.fr (#17705)
 * [philharmoniedeparis] Fix extraction (#17705)
 + [jamendo] Add support for licensing.jamendo.com (#17724)
 + [openload] Add support for oload.cloud (#17710)
 * [pluralsight] Fix subtitles extraction (#17726, #17728)
 + [vimeo] Add another config regular expression (#17690)
 * [spike] Fix Paramount Network extraction (#17677)
 * [hotstar] Fix extraction (#14694, #14931, #17637)
 version 2018.09.26
 Extractors
 * [pluralsight] Fix subtitles extraction (#17671)
 * [mediaset] Improve embed support (#17668)
 + [youtube] Add support for invidio.us (#17613)
 + [zattoo] Add support for more zattoo platform sites
 * [zattoo] Fix extraction (#17175, #17542)
 version 2018.09.18
 Core
 + [extractor/common] Introduce channel meta fields
 Extractors
 * [adobepass] Don't pollute default headers dict
 * [udemy] Don't pollute default headers dict
 * [twitch] Don't pollute default headers dict
 * [youtube] Don't pollute default query dict (#17593)
 * [crunchyroll] Prefer hardsubless formats and formats in locale language
 * [vrv] Make format ids deterministic
 * [vimeo] Fix ondemand playlist extraction (#14591)
 + [pornhub] Extract upload date (#17574)
 + [porntube] Extract channel meta fields
 + [vimeo] Extract channel meta fields
 + [youtube] Extract channel meta fields (#9676, #12939)
 * [porntube] Fix extraction (#17541)
 * [asiancrush] Fix extraction (#15630)
 + [twitch:clips] Extend URL regular expression (closes #17559)
 + [vzaar] Add support for HLS
 * [tube8] Fix metadata extraction (#17520)
 * [eporner] Extract JSON-LD (#17519)
 version 2018.09.10
 Core
 + [utils] Properly recognize AV1 codec (#17506)
 Extractors
 + [iprima] Add support for prima.iprima.cz (#17514)
 + [tele5] Add support for tele5.de (#7805, #7922, #17331, #17414)
 * [nbc] Fix extraction of percent encoded URLs (#17374)
 version 2018.09.08
 Extractors
 * [youtube] Fix extraction (#17457, #17464)
 + [pornhub:uservideos] Add support for new URLs (#17388)
 * [iprima] Confirm adult check (#17437)
 * [slideslive] Make check for video service name case-insensitive (#17429)
 * [radiojavan] Fix extraction (#17151)
 * [generic] Skip unsuccessful jwplayer extraction (#16735)
 version 2018.09.01
 Core
 * [utils] Skip remote IP addresses non matching to source address' IP version
  when creating a connection (#13422, #17362)
 Extractors
 + [ard] Add support for one.ard.de (#17397)
 * [niconico] Fix extraction on python3 (#17393, #17407)
 * [ard] Extract f4m formats
 * [crunchyroll] Parse vilos media data (#17343)
 + [ard] Add support for Beta ARD Mediathek
 + [bandcamp] Extract more metadata (#13197)
 * [internazionale] Fix extraction of non-available-abroad videos (#17386)
 version 2018.08.28
 Extractors
 + [youtube:playlist] Add support for music album playlists (OLAK5uy_ prefix)
  (#17361)
 * [bitchute] Fix extraction by pass custom User-Agent (#17360)
 * [webofstories:playlist] Fix extraction (#16914)
 + [tvplayhome] Add support for new tvplay URLs (#17344)
 + [generic] Allow relative src for videojs embeds (#17324)
 + [xfileshare] Add support for vidto.se (#17317)
 + [vidzi] Add support for vidzi.nu (#17316)
 + [nova:embed] Add support for media.cms.nova.cz (#17282)
 version 2018.08.22
 Core
 * [utils] Use pure browser header for User-Agent (#17236)
 Extractors
 + [kinopoisk] Add support for kinopoisk.ru (#17283)
 + [yourporn] Add support for yourporn.sexy (#17298)
 + [go] Add support for disneynow.go.com (#16299, #17264)
 + [6play] Add support for play.rtl.hr (#17249)
 * [anvato] Fallback to generic API key for access-key-to-API-key lookup
  (#16788, #17254)
 * [lci] Fix extraction (#17274)
 * [bbccouk] Extend id URL regular expression (#17270)
 * [cwtv] Fix extraction (#17256)
 * [nova] Fix extraction (#17241)
 + [generic] Add support for expressen embeds
 * [raywenderlich] Adapt to site redesign (#17225)
 + [redbulltv] Add support redbull.com tv URLs (#17218)
 + [bitchute] Add support for bitchute.com (#14052)
 + [clyp] Add support for token protected media (#17184)
 * [imdb] Fix extension extraction (#17167)
 version 2018.08.04
 Extractors
 * [funk:channel] Improve byChannelAlias extraction (#17142)
 * [twitch] Fix authentication (#17024, #17126)
 * [twitch:vod] Improve URL regular expression (#17135)
 * [watchbox] Fix extraction (#17107)
 * [pbs] Fix extraction (#17109)
 * [theplatform] Relax URL regular expression (#16181, #17097)
 + [viqeo] Add support for viqeo.tv (#17066)
 version 2018.07.29
 Extractors
 * [crunchyroll:playlist] Restrict URL regular expression (#17069, #17076)
 + [pornhub] Add support for subtitles (#16924, #17088)
 * [ceskatelevize] Use https for API call (#16997, #16999)
 * [dailymotion:playlist] Fix extraction (#16894)
 * [ted] Improve extraction
 * [ted] Fix extraction for videos without nativeDownloads (#16756, #17085)
 * [telecinco] Fix extraction (#17080)
 * [mitele] Reduce number of requests
 * [rai] Return non HTTP relinker URL intact (#17055)
 * [vk] Fix extraction for inline only videos (#16923)
 * [streamcloud] Fix extraction (#17054)
 * [facebook] Fix tahoe player extraction with authentication (#16655)
 + [puhutv] Add support for puhutv.com (#12712, #16010, #16269)
 version 2018.07.21
 Core
 + [utils] Introduce url_or_none
 * [utils] Allow JSONP without function name (#17028)
 + [extractor/common] Extract DASH and MSS formats from SMIL manifests
 Extractors
 + [bbc] Add support for BBC Radio Play pages (#17022)
 * [iwara] Fix download URLs (#17026)
 * [vrtnu] Relax title extraction and extract JSON-LD (#17018)
 + [viu] Pass Referer and Origin headers and area id (#16992)
 + [vimeo] Add another config regular expression (#17013)
 + [facebook] Extract view count (#16942)
 * [dailymotion] Improve description extraction (#16984)
 * [slutload] Fix and improve extraction (#17001)
 * [mediaset] Fix extraction (#16977)
 + [theplatform] Add support for theplatform TLD customization (#16977)
 * [imgur] Relax URL regular expression (#16987)
 * [pornhub] Improve extraction and extract all formats (#12166, #15891, #16262,
  #16959)
 version 2018.07.10
 Core
 * [utils] Share JSON-LD regular expression
 * [downloader/dash] Improve error handling (#16927)
 Extractors
 + [nrktv] Add support for new season and serie URL schema
 + [nrktv] Add support for new episode URL schema (#16909)
 + [frontendmasters] Add support for frontendmasters.com (#3661, #16328)
 * [funk] Fix extraction (#16918)
 * [watchbox] Fix extraction (#16904)
 * [dplayit] Sort formats
 * [dplayit] Fix extraction (#16901)
 * [youtube] Improve login error handling (#13822)
 version 2018.07.04
 Core
 * [extractor/common] Properly escape % in MPD templates (#16867)
 * [extractor/common] Use source URL as Referer for HTML5 entries (16849)
 * Prefer ffmpeg over avconv by default (#8622)
 Extractors
 * [pluralsight] Switch to graphql (#16889, #16895, #16896, #16899)
 * [lynda] Simplify login and improve error capturing (#16891)
 + [go90] Add support for embed URLs (#16873)
 * [go90] Detect geo restriction error and pass geo verification headers
  (#16874)
 * [vlive] Fix live streams extraction (#16871)
 * [npo] Fix typo (#16872)
 + [mediaset] Add support for new videos and extract all formats (#16568)
 * [dctptv] Restore extraction based on REST API (#16850)
 * [svt] Improve extraction and add support for pages (#16802)
 * [porncom] Fix extraction (#16808)
 version 2018.06.25
 Extractors
 * [joj] Relax URL regular expression (#16771)
 * [brightcove] Workaround sonyliv DRM protected videos (#16807)
 * [motherless] Fix extraction (#16786)
 * [itv] Make SOAP request non fatal and extract metadata from webpage (#16780)
 - [foxnews:insider] Remove extractor (#15810)
 + [foxnews] Add support for iframe embeds (#15810, #16711)
 version 2018.06.19
 Core
 + [extractor/common] Introduce expected_status in _download_* methods
  for convenient accept of HTTP requests failed with non 2xx status codes
 + [compat] Introduce compat_integer_types
 Extractors
 * [peertube] Improve generic support (#16733)
 + [6play] Use geo verification headers
 * [rtbf] Fix extraction for python 3.2
 * [vgtv] Improve HLS formats extraction
 + [vgtv] Add support for www.aftonbladet.se/tv URLs
 * [bbccouk] Use expected_status
 * [markiza] Expect 500 HTTP status code
 * [tvnow] Try all clear manifest URLs (#15361)
 version 2018.06.18
 Core
 * [downloader/rtmp] Fix downloading in verbose mode (#16736)
 Extractors
 + [markiza] Add support for markiza.sk (#16750)
 * [wat] Try all supported adaptive URLs
 + [6play] Add support for rtlplay.be and extract hd usp formats
 + [rtbf] Add support for audio and live streams (#9638, #11923)
 + [rtbf] Extract HLS, DASH and all HTTP formats
 + [rtbf] Extract subtitles
 + [rtbf] Fixup specific HTTP URLs (#16101)
 + [expressen] Add support for expressen.se
 * [vidzi] Fix extraction (#16678)
 * [pbs] Improve extraction (#16623, #16684)
 * [bilibili] Restrict cid regular expression (#16638, #16734)
 version 2018.06.14
 Core
 * [downloader/http] Fix retry on error when streaming to stdout (#16699)
 Extractors
 + [discoverynetworks] Add support for disco-api videos (#16724)
 + [dailymotion] Add support for password protected videos (#9789)
 + [abc:iview] Add support for livestreams (#12354)
 * [abc:iview] Fix extraction (#16704)
 + [crackle] Add support for sonycrackle.com (#16698)
 + [tvnet] Add support for tvnet.gov.vn (#15462)
 * [nrk] Update API hosts and try all previously known ones (#16690)
 * [wimp] Fix Youtube embeds extraction
 version 2018.06.11
 Extractors
 * [npo] Extend URL regular expression and add support for npostart.nl (#16682)
 + [inc] Add support for another embed schema (#16666)
 * [tv4] Fix format extraction (#16650)
 + [nexx] Add support for free cdn (#16538)
 + [pbs] Add another cove id pattern (#15373)
 + [rbmaradio] Add support for 192k format (#16631)
 version 2018.06.04
 Extractors
 + [camtube] Add support for camtube.co
 + [twitter:card] Extract guest token (#16609)
 + [chaturbate] Use geo verification headers
 + [bbc] Add support for bbcthree (#16612)
 * [youtube] Move metadata extraction after video availability check
 + [youtube] Extract track and artist
 + [safari] Add support for new URL schema (#16614)
 * [adn] Fix extraction
 version 2018.06.02
 Core
 * [utils] Improve determine_ext
 Extractors
 + [facebook] Add support for tahoe player videos (#15441, #16554)
 * [cbc] Improve extraction (#16583, #16593)
 * [openload] Improve ext extraction (#16595)
 + [twitter:card] Add support for another endpoint (#16586)
 + [openload] Add support for oload.win and oload.download (#16592)
 * [audimedia] Fix extraction (#15309)
 + [francetv] Add support for sport.francetvinfo.fr (#15645)
 * [mlb] Improve extraction (#16587)
 - [nhl] Remove old extractors
 * [rbmaradio] Check formats availability (#16585)
 version 2018.05.30
 Core
 * [downloader/rtmp] Generalize download messages and report time elapsed
  on finish
 * [downloader/rtmp] Gracefully handle live streams interrupted by user
 Extractors
 * [teamcoco] Fix extraction for full episodes (#16573)
 * [spiegel] Fix info extraction (#16538)
 + [apa] Add support for apa.at (#15041, #15672)
 + [bellmedia] Add support for bnnbloomberg.ca (#16560)
 + [9c9media] Extract MPD formats and subtitles
 * [cammodels] Use geo verification headers
 + [ufctv] Add support for authentication (#16542)
 + [cammodels] Add support for cammodels.com (#14499)
 * [utils] Fix style id extraction for namespaced id attribute in dfxp2srt
  (#16551)
 * [soundcloud] Detect format extension (#16549)
 * [cbc] Fix playlist title extraction (#16502)
 + [tumblr] Detect and report sensitive media (#13829)
 + [tumblr] Add support for authentication (#15133)
 version 2018.05.26
 Core
 * [utils] Improve parse_age_limit
 Extractors
 * [audiomack] Stringify video id (#15310)
 * [izlesene] Fix extraction (#16233, #16271, #16407)
 + [indavideo] Add support for generic embeds (#11989)
 * [indavideo] Fix extraction (#11221)
 * [indavideo] Sign download URLs (#16174)
 + [peertube] Add support for PeerTube based sites (#16301, #16329)
 * [imgur] Fix extraction (#16537)
 + [hidive] Add support for authentication (#16534)
 + [nbc] Add support for stream.nbcsports.com (#13911)
 + [viewlift] Add support for hoichoi.tv (#16536)
 * [go90] Extract age limit and detect DRM protection(#10127)
 * [viewlift] fix extraction for snagfilms.com (#15766)
 * [globo] Improve extraction (#4189)
    * Add support for authentication
    * Simplify URL signing
    * Extract DASH and MSS formats
 * [leeco] Fix extraction (#16464)
 * [teamcoco] Add fallback for format extraction (#16484)
 * [teamcoco] Improve URL regular expression (#16484)
 * [imdb] Improve extraction (#4085, #14557)
 version 2018.05.18
 Extractors
 * [vimeo:likes] Relax URL regular expression and fix single page likes
  extraction (#16475)
 * [pluralsight] Fix clip id extraction (#16460)
 + [mychannels] Add support for mychannels.com (#15334)
 - [moniker] Remove extractor (#15336)
 * [pbs] Fix embed data extraction (#16474)
 + [mtv] Add support for paramountnetwork.com and bellator.com (#15418)
 * [youtube] Fix hd720 format position
 * [dailymotion] Remove fragment part from m3u8 URLs (#8915)
 * [3sat] Improve extraction (#15350)
    * Extract all formats
    * Extract more format metadata
    * Improve format sorting
    * Use hls native downloader
    * Detect and bypass geo-restriction
 + [dtube] Add support for d.tube (#15201)
 * [options] Fix typo (#16450)
 * [youtube] Improve format filesize extraction (#16453)
 * [youtube] Make uploader extraction non fatal (#16444)
 * [youtube] Fix extraction for embed restricted live streams (#16433)
 * [nbc] Improve info extraction (#16440)
 * [twitch:clips] Fix extraction (#16429)
 * [redditr] Relax URL regular expression (#16426, #16427)
 * [mixcloud] Bypass throttling for HTTP formats (#12579, #16424)
 + [nick] Add support for nickjr.de (#13230)
 * [teamcoco] Fix extraction (#16374)
 version 2018.05.09
 Core
 * [YoutubeDL] Ensure ext exists for automatic captions
 * Introduce --geo-bypass-ip-block
 Extractors
 + [udemy] Extract asset captions
 + [udemy] Extract stream URLs (#16372)
 + [businessinsider] Add support for businessinsider.com (#16387, #16388, #16389)
 + [cloudflarestream] Add support for cloudflarestream.com (#16375)
 * [watchbox] Fix extraction (#16356)
 * [discovery] Extract Affiliate/Anonymous Auth Token from cookies (#14954)
 + [itv:btcc] Add support for itv.com/btcc (#16139)
 * [tunein] Use live title for live streams (#16347)
 * [itv] Improve extraction (#16253)
 version 2018.05.01
 Core
 * [downloader/fragment] Restart download if .ytdl file is corrupt (#16312)
 + [extractor/common] Extract interaction statistic
 + [utils] Add merge_dicts
 + [extractor/common] Add _download_json_handle
 Extractors
 * [kaltura] Improve iframe embeds detection (#16337)
 + [udemy] Extract outputs renditions (#16289, #16291, #16320, #16321, #16334,
  #16335)
 + [zattoo] Add support for zattoo.com and mobiltv.quickline.com (#14668, #14676)
 * [yandexmusic] Convert release_year to int
 * [udemy] Override _download_webpage_handle instead of _download_webpage
 * [xiami] Override _download_webpage_handle instead of _download_webpage
 * [yandexmusic] Override _download_webpage_handle instead of _download_webpage
 * [youtube] Correctly disable polymer on all requests (#16323, #16326)
 * [generic] Prefer enclosures over links in RSS feeds (#16189)
 + [redditr] Add support for old.reddit.com URLs (#16274)
 * [nrktv] Update API host (#16324)
 + [imdb] Extract all formats (#16249)
 + [vimeo] Extract JSON-LD (#16295)
 * [funk:channel] Improve extraction (#16285)
 version 2018.04.25
 Core
 * [utils] Fix match_str for boolean meta fields
 + [Makefile] Add support for pandoc 2 and disable smart extension (#16251)
 * [YoutubeDL] Fix typo in media extension compatibility checker (#16215)
 Extractors
 + [openload] Recognize IPv6 stream URLs (#16136, #16137, #16205, #16246,
  #16250)
 + [twitch] Extract is_live according to status (#16259)
 * [pornflip] Relax URL regular expression (#16258)
 - [etonline] Remove extractor (#16256)
 * [breakcom] Fix extraction (#16254)
 + [youtube] Add ability to authenticate with cookies
 * [youtube:feed] Implement lazy playlist extraction (#10184)
 + [svt] Add support for TV channel live streams (#15279, #15809)
 * [ccma] Fix video extraction (#15931)
 * [rentv] Fix extraction (#15227)
 + [nick] Add support for nickjr.nl (#16230)
 * [extremetube] Fix metadata extraction
 + [keezmovies] Add support for generic embeds (#16134, #16154)
 * [nexx] Extract new azure URLs (#16223)
 * [cbssports] Fix extraction (#16217)
 * [kaltura] Improve embeds detection (#16201)
 * [instagram:user] Fix extraction (#16119)
 * [cbs] Skip DRM asset types (#16104)
 version 2018.04.16
 Extractors
 * [smotri:broadcast] Fix extraction (#16180)
 + [picarto] Add support for picarto.tv (#6205, #12514, #15276, #15551)
 * [vine:user] Fix extraction (#15514, #16190)
 * [pornhub] Relax URL regular expression (#16165)
 * [cbc:watch] Re-acquire device token when expired (#16160)
 + [fxnetworks] Add support for https theplatform URLs (#16125, #16157)
 + [instagram:user] Add request signing (#16119)
 + [twitch] Add support for mobile URLs (#16146)
 version 2018.04.09
 Core
 * [YoutubeDL] Do not save/restore console title while simulate (#16103)
 * [extractor/common] Relax JSON-LD context check (#16006)
 Extractors
 + [generic] Add support for tube8 embeds
 + [generic] Add support for share-videos.se embeds (#16089, #16115)
 * [odnoklassniki] Extend URL regular expression (#16081)
 * [steam] Bypass mature content check (#16113)
 + [acast] Extract more metadata
 * [acast] Fix extraction (#16118)
 * [instagram:user] Fix extraction (#16119)
 * [drtuber] Fix title extraction (#16107, #16108)
 * [liveleak] Extend URL regular expression (#16117)
 + [openload] Add support for oload.xyz
 * [openload] Relax stream URL regular expression
 * [openload] Fix extraction (#16099)
 + [svtplay:series] Add support for season URLs
 + [svtplay:series] Add support for series (#11130, #16059)
 version 2018.04.03
 Extractors
 + [tvnow] Add support for shows (#15837)
 * [dramafever] Fix authentication (#16067)
 * [afreecatv] Use partial view only when necessary (#14450)
 + [afreecatv] Add support for authentication (#14450)
 + [nationalgeographic] Add support for new URL schema (#16001, #16054)
 * [xvideos] Fix thumbnail extraction (#15978, #15979)
 * [medialaan] Fix vod id (#16038)
 + [openload] Add support for oload.site (#16039)
 * [naver] Fix extraction (#16029)
 * [dramafever] Partially switch to API v5 (#16026)
 * [abc:iview] Unescape title and series meta fields (#15994)
 * [videa] Extend URL regular expression (#16003)
 version 2018.03.26.1
 Core
 + [downloader/external] Add elapsed time to progress hook (#10876)
 * [downloader/external,fragment] Fix download finalization when writing file
  to stdout (#10809, #10876, #15799)
 Extractors
 * [vrv] Fix extraction on python2 (#15928)
 * [afreecatv] Update referrer (#15947)
 + [24video] Add support for 24video.sexy (#15973)
 * [crackle] Bypass geo restriction
 * [crackle] Fix extraction (#15969)
 + [lenta] Add support for lenta.ru (#15953)
 + [instagram:user] Add pagination (#15934)
 * [youku] Update ccode (#15939)
 * [libsyn] Adapt to new page structure
 version 2018.03.20
 Core
--- a/7
+++ b/7
@ -14,6 +14,9 @@ PYTHON ?= /usr/bin/env python
 # set SYSCONFDIR to /etc if PREFIX=/usr or PREFIX=/usr/local
 SYSCONFDIR = $(shell if [ $(PREFIX) = /usr -o $(PREFIX) = /usr/local ]; then echo /etc; else echo $(PREFIX)/etc; fi)
 # set markdown input format to "markdown-smart" for pandoc version 2 and to "markdown" for pandoc prior to version 2
 MARKDOWN = $(shell if [ `pandoc -v | head -n1 | cut -d" " -f2 | head -c1` = "2" ]; then echo markdown-smart; else echo markdown; fi)
 install: youtube-dl youtube-dl.1 youtube-dl.bash-completion youtube-dl.zsh youtube-dl.fish
 	install -d $(DESTDIR)$(BINDIR)
 	install -m 755 youtube-dl $(DESTDIR)$(BINDIR)
@ -82,11 +85,11 @@ supportedsites:
 	$(PYTHON) devscripts/make_supportedsites.py docs/supportedsites.md
 README.txt: README.md
-	pandoc -f markdown -t plain README.md -o README.txt
+	pandoc -f $(MARKDOWN) -t plain README.md -o README.txt
 youtube-dl.1: README.md
 	$(PYTHON) devscripts/prepare_manpage.py youtube-dl.1.temp.md
-	pandoc -s -f markdown -t man youtube-dl.1.temp.md -o youtube-dl.1
+	pandoc -s -f $(MARKDOWN) -t man youtube-dl.1.temp.md -o youtube-dl.1
 	rm -f youtube-dl.1.temp.md
 youtube-dl.bash-completion: youtube_dl/*.py youtube_dl/*/*.py devscripts/bash-completion.in
--- a/README.md
+++ b/README.md
@ -17,7 +17,7 @@ youtube-dl - download videos from youtube.com or other video platforms
 # INSTALLATION
-To install it right away for all UNIX users (Linux, OS X, etc.), type:
+To install it right away for all UNIX users (Linux, macOS, etc.), type:
    sudo curl -L https://yt-dl.org/downloads/latest/youtube-dl -o /usr/local/bin/youtube-dl
    sudo chmod a+rx /usr/local/bin/youtube-dl
@ -35,7 +35,7 @@ You can also use pip:
 This command will update youtube-dl if you have already installed it. See the [pypi page](https://pypi.python.org/pypi/youtube_dl) for more information.
-OS X users can install youtube-dl with [Homebrew](https://brew.sh/):
+macOS users can install youtube-dl with [Homebrew](https://brew.sh/):
    brew install youtube-dl
@ -93,8 +93,8 @@ Alternatively, refer to the [developer instructions](#developer-instructions) fo
 ## Network Options:
    --proxy URL                      Use the specified HTTP/HTTPS/SOCKS proxy.
-                                     To enable experimental SOCKS proxy, specify
+                                     To enable SOCKS proxy, specify a proper
-                                     a proper scheme. For example
+                                     scheme. For example
                                     socks5://127.0.0.1:1080/. Pass in an empty
                                     string (--proxy "") for direct connection
    --socket-timeout SECONDS         Time to wait before giving up, in seconds
@ -106,16 +106,18 @@ Alternatively, refer to the [developer instructions](#developer-instructions) fo
    --geo-verification-proxy URL     Use this proxy to verify the IP address for
                                     some geo-restricted sites. The default
                                     proxy specified by --proxy (or none, if the
-                                     options is not present) is used for the
+                                     option is not present) is used for the
                                     actual downloading.
    --geo-bypass                     Bypass geographic restriction via faking
-                                     X-Forwarded-For HTTP header (experimental)
+                                     X-Forwarded-For HTTP header
    --no-geo-bypass                  Do not bypass geographic restriction via
                                     faking X-Forwarded-For HTTP header
                                     (experimental)
    --geo-bypass-country CODE        Force bypass geographic restriction with
                                     explicitly provided two-letter ISO 3166-2
-                                     country code (experimental)
+                                     country code
    --geo-bypass-ip-block IP_BLOCK   Force bypass geographic restriction with
                                     explicitly provided IP block in CIDR
                                     notation
 ## Video Selection:
    --playlist-start NUMBER          Playlist video to start at (default is 1)
@ -206,7 +208,7 @@ Alternatively, refer to the [developer instructions](#developer-instructions) fo
    --playlist-reverse               Download playlist videos in reverse order
    --playlist-random                Download playlist videos in random order
    --xattr-set-filesize             Set file xattribute ytdl.filesize with
-                                     expected file size (experimental)
+                                     expected file size
    --hls-prefer-native              Use the native HLS downloader instead of
                                     ffmpeg
    --hls-prefer-ffmpeg              Use ffmpeg instead of the native HLS
@ -223,7 +225,9 @@ Alternatively, refer to the [developer instructions](#developer-instructions) fo
 ## Filesystem Options:
    -a, --batch-file FILE            File containing URLs to download ('-' for
-                                     stdin)
+                                     stdin), one URL per line. Lines starting
                                     with '#', ';' or ']' are considered as
                                     comments and ignored.
    --id                             Use only video ID in file name
    -o, --output TEMPLATE            Output filename template, see the "OUTPUT
                                     TEMPLATE" for all the info
@ -423,9 +427,9 @@ Alternatively, refer to the [developer instructions](#developer-instructions) fo
                                     default; fix file if we can, warn
                                     otherwise)
    --prefer-avconv                  Prefer avconv over ffmpeg for running the
                                     postprocessors (default)
    --prefer-ffmpeg                  Prefer ffmpeg over avconv for running the
                                     postprocessors
    --prefer-ffmpeg                  Prefer ffmpeg over avconv for running the
                                     postprocessors (default)
    --ffmpeg-location PATH           Location of the ffmpeg/avconv binary;
                                     either the path to the binary or its
                                     containing directory.
@ -438,7 +442,7 @@ Alternatively, refer to the [developer instructions](#developer-instructions) fo
 # CONFIGURATION
-You can configure youtube-dl by placing any supported command line option to a configuration file. On Linux and OS X, the system wide configuration file is located at `/etc/youtube-dl.conf` and the user wide configuration file at `~/.config/youtube-dl/config`. On Windows, the user wide configuration file locations are `%APPDATA%\youtube-dl\config.txt` or `C:\Users\<user name>\youtube-dl.conf`. Note that by default configuration file may not exist so you may need to create it yourself.
+You can configure youtube-dl by placing any supported command line option to a configuration file. On Linux and macOS, the system wide configuration file is located at `/etc/youtube-dl.conf` and the user wide configuration file at `~/.config/youtube-dl/config`. On Windows, the user wide configuration file locations are `%APPDATA%\youtube-dl\config.txt` or `C:\Users\<user name>\youtube-dl.conf`. Note that by default configuration file may not exist so you may need to create it yourself.
 For example, with the following configuration file youtube-dl will always extract the audio, not copy the mtime, use a proxy and save all videos under `Movies` directory in your home directory:
 ```
@ -507,6 +511,8 @@ The basic usage is not to set any template arguments when downloading a single f
 - `timestamp` (numeric): UNIX timestamp of the moment the video became available
 - `upload_date` (string): Video upload date (YYYYMMDD)
 - `uploader_id` (string): Nickname or id of the video uploader
 - `channel` (string): Full name of the channel the video is uploaded on
 - `channel_id` (string): Id of the channel
 - `location` (string): Physical location where the video was filmed
 - `duration` (numeric): Length of the video in seconds
 - `view_count` (numeric): How many users have watched the video on the platform
@ -866,7 +872,7 @@ Either prepend `https://www.youtube.com/watch?v=` or separate the ID from the op
 Use the `--cookies` option, for example `--cookies /path/to/cookies/file.txt`.
-In order to extract cookies from browser use any conforming browser extension for exporting cookies. For example, [cookies.txt](https://chrome.google.com/webstore/detail/cookiestxt/njabckikapfpffapmjgojcnbfjonfjfg) (for Chrome) or [Export Cookies](https://addons.mozilla.org/en-US/firefox/addon/export-cookies/) (for Firefox).
+In order to extract cookies from browser use any conforming browser extension for exporting cookies. For example, [cookies.txt](https://chrome.google.com/webstore/detail/cookiestxt/njabckikapfpffapmjgojcnbfjonfjfg) (for Chrome) or [cookies.txt](https://addons.mozilla.org/en-US/firefox/addon/cookies-txt/) (for Firefox).
 Note that the cookies file must be in Mozilla/Netscape format and the first line of the cookies file must be either `# HTTP Cookie File` or `# Netscape HTTP Cookie File`. Make sure you have correct [newline format](https://en.wikipedia.org/wiki/Newline) in the cookies file and convert newlines if necessary to correspond with your OS, namely `CRLF` (`\r\n`) for Windows and `LF` (`\n`) for Unix and Unix-like systems (Linux, macOS, etc.). `HTTP Error 400: Bad Request` when using `--cookies` is a good sign of invalid newline format.
--- a/devscripts/gh-pages/generate-download.py
+++ b/devscripts/gh-pages/generate-download.py
@ -1,27 +1,22 @@
 #!/usr/bin/env python3
 from __future__ import unicode_literals
 import hashlib
 import urllib.request
 import json
 versions_info = json.load(open('update/versions.json'))
 version = versions_info['latest']
-URL = versions_info['versions'][version]['bin'][0]
+version_dict = versions_info['versions'][version]
 data = urllib.request.urlopen(URL).read()
 # Read template page
 with open('download.html.in', 'r', encoding='utf-8') as tmplf:
    template = tmplf.read()
 sha256sum = hashlib.sha256(data).hexdigest()
 template = template.replace('@PROGRAM_VERSION@', version)
-template = template.replace('@PROGRAM_URL@', URL)
+template = template.replace('@PROGRAM_URL@', version_dict['bin'][0])
-template = template.replace('@PROGRAM_SHA256SUM@', sha256sum)
+template = template.replace('@PROGRAM_SHA256SUM@', version_dict['bin'][1])
-template = template.replace('@EXE_URL@', versions_info['versions'][version]['exe'][0])
+template = template.replace('@EXE_URL@', version_dict['exe'][0])
-template = template.replace('@EXE_SHA256SUM@', versions_info['versions'][version]['exe'][1])
+template = template.replace('@EXE_SHA256SUM@', version_dict['exe'][1])
-template = template.replace('@TAR_URL@', versions_info['versions'][version]['tar'][0])
+template = template.replace('@TAR_URL@', version_dict['tar'][0])
-template = template.replace('@TAR_SHA256SUM@', versions_info['versions'][version]['tar'][1])
+template = template.replace('@TAR_SHA256SUM@', version_dict['tar'][1])
 with open('download.html', 'w', encoding='utf-8') as dlf:
    dlf.write(template)
--- a/devscripts/gh-pages/update-copyright.py
+++ b/devscripts/gh-pages/update-copyright.py
@ -13,7 +13,7 @@ year = str(datetime.datetime.now().year)
 for fn in glob.glob('*.html*'):
    with io.open(fn, encoding='utf-8') as f:
        content = f.read()
-    newc = re.sub(r'(?P<copyright>Copyright © 2006-)(?P<year>[0-9]{4})', 'Copyright © 2006-' + year, content)
+    newc = re.sub(r'(?P<copyright>Copyright © 2011-)(?P<year>[0-9]{4})', 'Copyright © 2011-' + year, content)
    if content != newc:
        tmpFn = fn + '.part'
        with io.open(tmpFn, 'wt', encoding='utf-8') as outf:
--- a/docs/supportedsites.md
+++ b/docs/supportedsites.md
@ -15,7 +15,6 @@
 - **8tracks**
 - **91porn**
 - **9c9media**
 - **9c9media:stack**
 - **9gag**
 - **9now.com.au**
 - **abc.net.au**
@ -48,6 +47,7 @@
 - **anitube.se**
 - **Anvato**
 - **AnySex**
 - **APA**
 - **Aparat**
 - **AppleConnect**
 - **AppleDaily**: 臺灣蘋果日報
@ -56,6 +56,7 @@
 - **archive.org**: archive.org videos
 - **ARD**
 - **ARD:mediathek**
 - **ARDBetaMediathek**
 - **Arkena**
 - **arte.tv**
 - **arte.tv:+7**
@ -97,9 +98,11 @@
 - **bbc.co.uk:article**: BBC articles
 - **bbc.co.uk:iplayer:playlist**
 - **bbc.co.uk:playlist**
 - **BBVTV**
 - **Beatport**
 - **Beeg**
 - **BehindKink**
 - **Bellator**
 - **BellMedia**
 - **Bet**
 - **Bigflix**
@ -107,6 +110,8 @@
 - **BiliBili**
 - **BioBioChileTV**
 - **BIQLE**
 - **BitChute**
 - **BitChuteChannel**
 - **BleacherReport**
 - **BleacherReportCMS**
 - **blinkx**
@ -122,10 +127,13 @@
 - **BRMediathek**: Bayerischer Rundfunk Mediathek
 - **bt:article**: Bergens Tidende Articles
 - **bt:vestlendingen**: Bergens Tidende - Vestlendingen
 - **BusinessInsider**
 - **BuzzFeed**
 - **BYUtv**
 - **Camdemy**
 - **CamdemyFolder**
 - **CamModels**
 - **CamTube**
 - **CamWithHer**
 - **canalc2.tv**
 - **Canalplus**: mycanal.fr and piwiplus.fr
@ -163,6 +171,7 @@
 - **ClipRs**
 - **Clipsyndicate**
 - **CloserToTruth**
 - **CloudflareStream**
 - **cloudtime**: CloudTime
 - **Cloudy**
 - **Clubic**
@ -184,7 +193,7 @@
 - **Crackle**
 - **Criterion**
 - **CrooksAndLiars**
- - **Crunchyroll**
+ - **crunchyroll**
 - **crunchyroll:playlist**
 - **CSNNE**
 - **CSpan**: C-SPAN
@ -232,6 +241,7 @@
 - **DrTuber**
 - **drtv**
 - **drtv:live**
 - **DTube**
 - **Dumpert**
 - **dvtv**: http://video.aktualne.cz/
 - **dw**
@ -242,6 +252,7 @@
 - **egghead:course**: egghead.io course
 - **egghead:lesson**: egghead.io lesson
 - **eHow**
 - **EinsUndEinsTV**
 - **Einthusan**
 - **eitb.tv**
 - **EllenTube**
@ -257,10 +268,11 @@
 - **ESPN**
 - **ESPNArticle**
 - **EsriVideo**
 - **ETOnline**
 - **Europa**
 - **EveryonesMixtape**
 - **EWETV**
 - **ExpoTV**
 - **Expressen**
 - **ExtremeTube**
 - **EyedoTV**
 - **facebook**
@ -284,7 +296,6 @@
 - **Foxgay**
 - **foxnews**: Fox News and Fox Business Video
 - **foxnews:article**
 - **foxnews:insider**
 - **FoxSports**
 - **france2.fr:generation-what**
 - **FranceCulture**
@ -297,6 +308,9 @@
 - **Freesound**
 - **freespeech.org**
 - **FreshLive**
 - **FrontendMasters**
 - **FrontendMastersCourse**
 - **FrontendMastersLesson**
 - **Funimation**
 - **FunkChannel**
 - **FunkMix**
@ -316,6 +330,7 @@
 - **Gfycat**
 - **GiantBomb**
 - **Giga**
 - **GlattvisionTV**
 - **Glide**: Glide mobile video messages (glide.me)
 - **Globo**
 - **GloboArticle**
@ -345,7 +360,7 @@
 - **HitRecord**
 - **HornBunny**
 - **HotNewHipHop**
- - **HotStar**
+ - **hotstar**
 - **hotstar:playlist**
 - **Howcast**
 - **HowStuffWorks**
@ -362,7 +377,6 @@
 - **ImgurAlbum**
 - **Ina**
 - **Inc**
 - **Indavideo**
 - **IndavideoEmbed**
 - **InfoQ**
 - **Instagram**
@ -374,6 +388,7 @@
 - **Ir90Tv**
 - **ITTF**
 - **ITV**
 - **ITVBTCC**
 - **ivi**: ivi.ru
 - **ivi:compilation**: ivi.ru compilations
 - **ivideon**: Ivideon TV
@ -397,6 +412,7 @@
 - **Ketnet**
 - **KhanAcademy**
 - **KickStarter**
 - **KinoPoisk**
 - **KonserthusetPlay**
 - **kontrtube**: KontrTube.ru - Труба зовёт
 - **KrasView**: Красвью
@ -419,6 +435,7 @@
 - **Lecture2Go**
 - **LEGO**
 - **Lemonde**
 - **Lenta**
 - **LePlaylist**
 - **LetvCloud**: 乐视云
 - **Libsyn**
@ -445,11 +462,12 @@
 - **mailru**: Видео@Mail.Ru
 - **mailru:music**: Музыка@Mail.Ru
 - **mailru:music:search**: Музыка@Mail.Ru
 - **MakersChannel**
 - **MakerTV**
 - **mangomolo:live**
 - **mangomolo:video**
 - **ManyVids**
 - **Markiza**
 - **MarkizaPage**
 - **massengeschmack.tv**
 - **MatchTV**
 - **MDR**: MDR.DE and KiKA
@ -480,10 +498,10 @@
 - **Mixer:vod**
 - **MLB**
 - **Mnet**
 - **MNetTV**
 - **MoeVideo**: LetitBit video services: moevideo.net, playreplay.net and videochart.net
 - **Mofosex**
 - **Mojvideo**
 - **Moniker**: allmyvideos.net and vidspot.net
 - **Morningstar**: morningstar.com
 - **Motherless**
 - **MotherlessGroup**
@ -505,12 +523,14 @@
 - **mva:course**: Microsoft Virtual Academy courses
 - **Mwave**
 - **MwaveMeetGreet**
 - **MyChannels**
 - **MySpace**
 - **MySpace:album**
 - **MySpass**
 - **Myvi**
 - **MyVidster**
 - **MyviEmbed**
 - **MyVisionTV**
 - **n-tv.de**
 - **natgeo**
 - **natgeo:episodeguide**
@ -522,6 +542,7 @@
 - **nbcolympics**
 - **nbcolympics:stream**
 - **NBCSports**
 - **NBCSportsStream**
 - **NBCSportsVPlayer**
 - **ndr**: NDR.de - Norddeutscher Rundfunk
 - **ndr:embed**
@ -535,6 +556,7 @@
 - **netease:program**: 网易云音乐 - 电台节目
 - **netease:singer**: 网易云音乐 - 歌手
 - **netease:song**: 网易云音乐
 - **NetPlus**
 - **Netzkino**
 - **Newgrounds**
 - **NewgroundsPlaylist**
@ -548,9 +570,6 @@
 - **nfl.com**
 - **NhkVod**
 - **nhl.com**
 - **nhl.com:news**: NHL news
 - **nhl.com:videocenter**
 - **nhl.com:videocenter:category**: NHL videocenter category
 - **nick.com**
 - **nick.de**
 - **nickelodeon:br**
@ -569,6 +588,7 @@
 - **Normalboots**
 - **NosVideo**
 - **Nova**: TN.cz, Prásk.tv, Nova.cz, Novaplus.cz, FANDA.tv, Krásná.cz and Doma.cz
 - **NovaEmbed**
 - **nowness**
 - **nowness:playlist**
 - **nowness:series**
@ -584,7 +604,9 @@
 - **NRKSkole**: NRK Skole
 - **NRKTV**: NRK TV and NRK Radio
 - **NRKTVDirekte**: NRK TV Direkte and NRK Radio Direkte
 - **NRKTVEpisode**
 - **NRKTVEpisodes**
 - **NRKTVSeason**
 - **NRKTVSeries**
 - **ntv.ru**
 - **Nuvid**
@ -611,15 +633,18 @@
 - **orf:iptv**: iptv.ORF.at
 - **orf:oe1**: Radio Österreich 1
 - **orf:tvthek**: ORF TVthek
 - **OsnatelTV**
 - **PacktPub**
 - **PacktPubCourse**
 - **PandaTV**: 熊猫TV
 - **pandora.tv**: 판도라TV
 - **ParamountNetwork**
 - **parliamentlive.tv**: UK parliament videos
 - **Patreon**
 - **pbs**: Public Broadcasting Service (PBS) and member stations: PBS: Public Broadcasting Service, APT - Alabama Public Television (WBIQ), GPB/Georgia Public Broadcasting (WGTV), Mississippi Public Broadcasting (WMPN), Nashville Public Television (WNPT), WFSU-TV (WFSU), WSRE (WSRE), WTCI (WTCI), WPBA/Channel 30 (WPBA), Alaska Public Media (KAKM), Arizona PBS (KAET), KNME-TV/Channel 5 (KNME), Vegas PBS (KLVX), AETN/ARKANSAS ETV NETWORK (KETS), KET (WKLE), WKNO/Channel 10 (WKNO), LPB/LOUISIANA PUBLIC BROADCASTING (WLPB), OETA (KETA), Ozarks Public Television (KOZK), WSIU Public Broadcasting (WSIU), KEET TV (KEET), KIXE/Channel 9 (KIXE), KPBS San Diego (KPBS), KQED (KQED), KVIE Public Television (KVIE), PBS SoCal/KOCE (KOCE), ValleyPBS (KVPT), CONNECTICUT PUBLIC TELEVISION (WEDH), KNPB Channel 5 (KNPB), SOPTV (KSYS), Rocky Mountain PBS (KRMA), KENW-TV3 (KENW), KUED Channel 7 (KUED), Wyoming PBS (KCWC), Colorado Public Television / KBDI 12 (KBDI), KBYU-TV (KBYU), Thirteen/WNET New York (WNET), WGBH/Channel 2 (WGBH), WGBY (WGBY), NJTV Public Media NJ (WNJT), WLIW21 (WLIW), mpt/Maryland Public Television (WMPB), WETA Television and Radio (WETA), WHYY (WHYY), PBS 39 (WLVT), WVPT - Your Source for PBS and More! (WVPT), Howard University Television (WHUT), WEDU PBS (WEDU), WGCU Public Media (WGCU), WPBT2 (WPBT), WUCF TV (WUCF), WUFT/Channel 5 (WUFT), WXEL/Channel 42 (WXEL), WLRN/Channel 17 (WLRN), WUSF Public Broadcasting (WUSF), ETV (WRLK), UNC-TV (WUNC), PBS Hawaii - Oceanic Cable Channel 10 (KHET), Idaho Public Television (KAID), KSPS (KSPS), OPB (KOPB), KWSU/Channel 10 & KTNW/Channel 31 (KWSU), WILL-TV (WILL), Network Knowledge - WSEC/Springfield (WSEC), WTTW11 (WTTW), Iowa Public Television/IPTV (KDIN), Nine Network (KETC), PBS39 Fort Wayne (WFWA), WFYI Indianapolis (WFYI), Milwaukee Public Television (WMVS), WNIN (WNIN), WNIT Public Television (WNIT), WPT (WPNE), WVUT/Channel 22 (WVUT), WEIU/Channel 51 (WEIU), WQPT-TV (WQPT), WYCC PBS Chicago (WYCC), WIPB-TV (WIPB), WTIU (WTIU), CET  (WCET), ThinkTVNetwork (WPTD), WBGU-TV (WBGU), WGVU TV (WGVU), NET1 (KUON), Pioneer Public Television (KWCM), SDPB Television (KUSD), TPT (KTCA), KSMQ (KSMQ), KPTS/Channel 8 (KPTS), KTWU/Channel 11 (KTWU), East Tennessee PBS (WSJK), WCTE-TV (WCTE), WLJT, Channel 11 (WLJT), WOSU TV (WOSU), WOUB/WOUC (WOUB), WVPB (WVPB), WKYU-PBS (WKYU), KERA 13 (KERA), MPBN (WCBB), Mountain Lake PBS (WCFE), NHPTV (WENH), Vermont PBS (WETK), witf (WITF), WQED Multimedia (WQED), WMHT Educational Telecommunications (WMHT), Q-TV (WDCQ), WTVS Detroit Public TV (WTVS), CMU Public Television (WCMU), WKAR-TV (WKAR), WNMU-TV Public TV 13 (WNMU), WDSE - WRPT (WDSE), WGTE TV (WGTE), Lakeland Public Television (KAWE), KMOS-TV - Channels 6.1, 6.2 and 6.3 (KMOS), MontanaPBS (KUSM), KRWG/Channel 22 (KRWG), KACV (KACV), KCOS/Channel 13 (KCOS), WCNY/Channel 24 (WCNY), WNED (WNED), WPBS (WPBS), WSKG Public TV (WSKG), WXXI (WXXI), WPSU (WPSU), WVIA Public Media Studios (WVIA), WTVI (WTVI), Western Reserve PBS (WNEO), WVIZ/PBS ideastream (WVIZ), KCTS 9 (KCTS), Basin PBS (KPBT), KUHT / Channel 8 (KUHT), KLRN (KLRN), KLRU (KLRU), WTJX Channel 12 (WTJX), WCVE PBS (WCVE), KBTC Public Television (KBTC)
 - **pcmag**
 - **PearVideo**
 - **PeerTube**
 - **People**
 - **PerformGroup**
 - **periscope**: Periscope
@ -627,6 +652,8 @@
 - **PhilharmonieDeParis**: Philharmonie de Paris
 - **phoenix.de**
 - **Photobucket**
 - **Picarto**
 - **PicartoVod**
 - **Piksel**
 - **Pinkbike**
 - **Pladform**
@ -658,6 +685,8 @@
 - **PrimeShareTV**
 - **PromptFile**
 - **prosiebensat1**: ProSiebenSat.1 Digital
 - **puhutv**
 - **puhutv:serie**
 - **Puls4**
 - **Pyvideo**
 - **qqmusic**: QQ音乐
@ -665,6 +694,9 @@
 - **qqmusic:playlist**: QQ音乐 - 歌单
 - **qqmusic:singer**: QQ音乐 - 歌手
 - **qqmusic:toplist**: QQ音乐 - 排行榜
 - **QuantumTV**
 - **Quickline**
 - **QuicklineLive**
 - **R7**
 - **R7Article**
 - **radio.de**
@ -678,6 +710,7 @@
 - **RaiPlayLive**
 - **RaiPlayPlaylist**
 - **RayWenderlich**
 - **RayWenderlichCourse**
 - **RBMARadio**
 - **RDS**: RDS.ca
 - **RedBullTV**
@ -729,6 +762,7 @@
 - **safari**: safaribooksonline.com online video
 - **safari:api**
 - **safari:course**: safaribooksonline.com online courses
 - **SAKTV**
 - **Sapo**: SAPO Vídeos
 - **savefrom.net**
 - **SBS**: sbs.com.au
@ -782,7 +816,7 @@
 - **Spiegel**
 - **Spiegel:Article**: Articles on spiegel.de
 - **Spiegeltv**
- - **Spike**
+ - **sport.francetvinfo.fr**
 - **Sport5**
 - **SportBoxEmbed**
 - **SportDeutschland**
@ -802,7 +836,9 @@
 - **StretchInternet**
 - **SunPorno**
 - **SVT**
 - **SVTPage**
 - **SVTPlay**: SVT Play and Öppet arkiv
 - **SVTSeries**
 - **SWRMediathek**
 - **Syfy**
 - **SztvHu**
@ -821,6 +857,7 @@
 - **techtv.mit.edu**
 - **ted**
 - **Tele13**
 - **Tele5**
 - **TeleBruxelles**
 - **Telecinco**: telecinco.es, cuatro.com and mediaset.es
 - **Telegraaf**
@ -883,13 +920,16 @@
 - **tvigle**: Интернет-телевидение Tvigle.ru
 - **tvland.com**
 - **TVN24**
 - **TVNet**
 - **TVNoe**
 - **TVNow**
 - **TVNowList**
 - **TVNowShow**
 - **tvp**: Telewizja Polska
 - **tvp:embed**: Telewizja Polska
 - **tvp:series**
 - **TVPlayer**
 - **TVPlayHome**
 - **Tweakers**
 - **twitch:chapter**
 - **twitch:clips**
@ -979,6 +1019,7 @@
 - **Vimple**: Vimple - one-click video hosting
 - **Vine**
 - **vine:user**
 - **Viqeo**
 - **Viu**
 - **viu:ott**
 - **viu:playlist**
@ -1004,12 +1045,14 @@
 - **vrv**
 - **vrv:series**
 - **VShare**
 - **VTXTV**
 - **vube**: Vube.com
 - **VuClip**
 - **VVVVID**
 - **VyboryMos**
 - **Vzaar**
 - **Walla**
 - **WalyTV**
 - **washingtonpost**
 - **washingtonpost:article**
 - **wat.tv**
@ -1070,6 +1113,7 @@
 - **YouNowLive**
 - **YouNowMoment**
 - **YouPorn**
 - **YourPorn**
 - **YourUpload**
 - **youtube**: YouTube.com
 - **youtube:channel**: YouTube.com channels
@ -1088,6 +1132,8 @@
 - **youtube:watchlater**: Youtube watch later list, ":ytwatchlater" for short (requires authentication)
 - **Zapiks**
 - **Zaq1**
 - **Zattoo**
 - **ZattooLive**
 - **ZDF**
 - **ZDFChannel**
 - **zingmp3**: mp3.zing.vn
--- a/setup.cfg
+++ b/setup.cfg
@ -2,5 +2,5 @@
 universal = True
 [flake8]
-exclude = youtube_dl/extractor/__init__.py,devscripts/buildserver.py,devscripts/lazy_load_template.py,devscripts/make_issue_template.py,setup.py,build,.git
+exclude = youtube_dl/extractor/__init__.py,devscripts/buildserver.py,devscripts/lazy_load_template.py,devscripts/make_issue_template.py,setup.py,build,.git,venv
 ignore = E402,E501,E731,E741
--- a/test/test_subtitles.py
+++ b/test/test_subtitles.py
@ -232,7 +232,7 @@ class TestNPOSubtitles(BaseTestSubtitles):
 class TestMTVSubtitles(BaseTestSubtitles):
-    url = 'http://www.cc.com/video-clips/kllhuv/stand-up-greg-fitzsimmons--uncensored---too-good-of-a-mother'
+    url = 'http://www.cc.com/video-clips/p63lk0/adam-devine-s-house-party-chasing-white-swans'
    IE = ComedyCentralIE
    def getInfoDict(self):
@ -243,7 +243,7 @@ class TestMTVSubtitles(BaseTestSubtitles):
        self.DL.params['allsubtitles'] = True
        subtitles = self.getSubtitles()
        self.assertEqual(set(subtitles.keys()), set(['en']))
-        self.assertEqual(md5(subtitles['en']), 'b9f6ca22a6acf597ec76f61749765e65')
+        self.assertEqual(md5(subtitles['en']), '78206b8d8a0cfa9da64dc026eea48961')
 class TestNRKSubtitles(BaseTestSubtitles):
--- a/test/test_utils.py
+++ b/test/test_utils.py
@ -42,6 +42,7 @@ from youtube_dl.utils import (
    is_html,
    js_to_json,
    limit_length,
    merge_dicts,
    mimetype2ext,
    month_by_name,
    multipart_encode,
@ -77,6 +78,7 @@ from youtube_dl.utils import (
    uppercase_escape,
    lowercase_escape,
    url_basename,
    url_or_none,
    base_url,
    urljoin,
    urlencode_postdata,
@ -360,6 +362,7 @@ class TestUtil(unittest.TestCase):
        self.assertEqual(determine_ext('http://example.com/foo/bar.nonext/?download', None), None)
        self.assertEqual(determine_ext('http://example.com/foo/bar/mp4?download', None), None)
        self.assertEqual(determine_ext('http://example.com/foo/bar.m3u8//?download'), 'm3u8')
        self.assertEqual(determine_ext('foobar', None), None)
    def test_find_xpath_attr(self):
        testxml = '''<root>
@ -505,6 +508,16 @@ class TestUtil(unittest.TestCase):
        self.assertEqual(urljoin('http://foo.de/', ['foobar']), None)
        self.assertEqual(urljoin('http://foo.de/a/b/c.txt', '.././../d.txt'), 'http://foo.de/d.txt')
    def test_url_or_none(self):
        self.assertEqual(url_or_none(None), None)
        self.assertEqual(url_or_none(''), None)
        self.assertEqual(url_or_none('foo'), None)
        self.assertEqual(url_or_none('http://foo.de'), 'http://foo.de')
        self.assertEqual(url_or_none('https://foo.de'), 'https://foo.de')
        self.assertEqual(url_or_none('http$://foo.de'), None)
        self.assertEqual(url_or_none('http://foo.de'), 'http://foo.de')
        self.assertEqual(url_or_none('//foo.de'), '//foo.de')
    def test_parse_age_limit(self):
        self.assertEqual(parse_age_limit(None), None)
        self.assertEqual(parse_age_limit(False), None)
@ -518,6 +531,8 @@ class TestUtil(unittest.TestCase):
        self.assertEqual(parse_age_limit('PG-13'), 13)
        self.assertEqual(parse_age_limit('TV-14'), 14)
        self.assertEqual(parse_age_limit('TV-MA'), 17)
        self.assertEqual(parse_age_limit('TV14'), 14)
        self.assertEqual(parse_age_limit('TV_G'), 0)
    def test_parse_duration(self):
        self.assertEqual(parse_duration(None), None)
@ -669,6 +684,17 @@ class TestUtil(unittest.TestCase):
            self.assertEqual(dict_get(d, ('b', 'c', key, )), None)
            self.assertEqual(dict_get(d, ('b', 'c', key, ), skip_false_values=False), false_value)
    def test_merge_dicts(self):
        self.assertEqual(merge_dicts({'a': 1}, {'b': 2}), {'a': 1, 'b': 2})
        self.assertEqual(merge_dicts({'a': 1}, {'a': 2}), {'a': 1})
        self.assertEqual(merge_dicts({'a': 1}, {'a': None}), {'a': 1})
        self.assertEqual(merge_dicts({'a': 1}, {'a': ''}), {'a': 1})
        self.assertEqual(merge_dicts({'a': 1}, {}), {'a': 1})
        self.assertEqual(merge_dicts({'a': None}, {'a': 1}), {'a': 1})
        self.assertEqual(merge_dicts({'a': ''}, {'a': 1}), {'a': ''})
        self.assertEqual(merge_dicts({'a': ''}, {'a': 'abc'}), {'a': 'abc'})
        self.assertEqual(merge_dicts({'a': None}, {'a': ''}, {'a': 'abc'}), {'a': 'abc'})
    def test_encode_compat_str(self):
        self.assertEqual(encode_compat_str(b'\xd1\x82\xd0\xb5\xd1\x81\xd1\x82', 'utf-8'), 'тест')
        self.assertEqual(encode_compat_str('тест', 'utf-8'), 'тест')
@ -702,6 +728,10 @@ class TestUtil(unittest.TestCase):
        d = json.loads(stripped)
        self.assertEqual(d, {'status': 'success'})
        stripped = strip_jsonp('({"status": "success"});')
        d = json.loads(stripped)
        self.assertEqual(d, {'status': 'success'})
    def test_uppercase_escape(self):
        self.assertEqual(uppercase_escape('aä'), 'aä')
        self.assertEqual(uppercase_escape('\\U0001d550'), '𝕐')
@ -755,6 +785,10 @@ class TestUtil(unittest.TestCase):
            'vcodec': 'h264',
            'acodec': 'aac',
        })
        self.assertEqual(parse_codecs('av01.0.05M.08'), {
            'vcodec': 'av01.0.05M.08',
            'acodec': 'none',
        })
    def test_escape_rfc3986(self):
        reserved = "!*'();:@&=+$,/?#[]"
@ -1072,6 +1106,18 @@ ffmpeg version 2.4.4 Copyright (c) 2000-2014 the FFmpeg ...'''), '2.4.4')
        self.assertFalse(match_str(
            'like_count > 100 & dislike_count <? 50 & description',
            {'like_count': 190, 'dislike_count': 10}))
        self.assertTrue(match_str('is_live', {'is_live': True}))
        self.assertFalse(match_str('is_live', {'is_live': False}))
        self.assertFalse(match_str('is_live', {'is_live': None}))
        self.assertFalse(match_str('is_live', {}))
        self.assertFalse(match_str('!is_live', {'is_live': True}))
        self.assertTrue(match_str('!is_live', {'is_live': False}))
        self.assertTrue(match_str('!is_live', {'is_live': None}))
        self.assertTrue(match_str('!is_live', {}))
        self.assertTrue(match_str('title', {'title': 'abc'}))
        self.assertTrue(match_str('title', {'title': ''}))
        self.assertFalse(match_str('!title', {'title': 'abc'}))
        self.assertFalse(match_str('!title', {'title': ''}))
    def test_parse_dfxp_time_expr(self):
        self.assertEqual(parse_dfxp_time_expr(None), None)
--- a/test/test_youtube_lists.py
+++ b/test/test_youtube_lists.py
@ -61,7 +61,7 @@ class TestYoutubeLists(unittest.TestCase):
        dl = FakeYDL()
        dl.params['extract_flat'] = True
        ie = YoutubePlaylistIE(dl)
-        result = ie.extract('https://www.youtube.com/playlist?list=PLwiyx1dc3P2JR9N8gQaQN_BCvlSlap7re')
+        result = ie.extract('https://www.youtube.com/playlist?list=PL-KKIb8rvtMSrAO9YFbeM6UQrAqoFTUWv')
        self.assertIsPlaylist(result)
        for entry in result['entries']:
            self.assertTrue(entry.get('title'))
--- a/youtube_dl/YoutubeDL.py
+++ b/youtube_dl/YoutubeDL.py
@ -211,7 +211,7 @@ class YoutubeDL(object):
                       At the moment, this is only supported by YouTube.
    proxy:             URL of the proxy server to use
    geo_verification_proxy:  URL of the proxy to use for IP address verification
-                       on geo-restricted sites. (Experimental)
+                       on geo-restricted sites.
    socket_timeout:    Time to wait for unresponsive hosts, in seconds
    bidi_workaround:   Work around buggy terminals without bidirectional text
                       support, using fridibi
@ -259,7 +259,7 @@ class YoutubeDL(object):
                       - "warn": only emit a warning
                       - "detect_or_warn": check whether we can do anything
                                           about it, warn otherwise (default)
-    source_address:    (Experimental) Client-side IP address to bind to.
+    source_address:    Client-side IP address to bind to.
    call_home:         Boolean, true iff we are allowed to contact the
                       youtube-dl servers for debugging.
    sleep_interval:    Number of seconds to sleep before each download when
@ -281,11 +281,14 @@ class YoutubeDL(object):
                       match_filter_func in utils.py is one example for this.
    no_color:          Do not emit color codes in output.
    geo_bypass:        Bypass geographic restriction via faking X-Forwarded-For
-                       HTTP header (experimental)
+                       HTTP header
    geo_bypass_country:
                       Two-letter ISO 3166-2 country code that will be used for
                       explicit geographic restriction bypassing via faking
-                       X-Forwarded-For HTTP header (experimental)
+                       X-Forwarded-For HTTP header
    geo_bypass_ip_block:
                       IP range in CIDR notation that will be used similarly to
                       geo_bypass_country
    The following options determine which downloader is picked:
    external_downloader: Executable of the external downloader to call.
@ -302,8 +305,8 @@ class YoutubeDL(object):
    http_chunk_size.
    The following options are used by the post processors:
-    prefer_ffmpeg:     If True, use ffmpeg instead of avconv if both are available,
+    prefer_ffmpeg:     If False, use avconv instead of ffmpeg if both are available,
-                       otherwise prefer avconv.
+                       otherwise prefer ffmpeg.
    postprocessor_args: A list of additional command-line arguments for the
                        postprocessor.
@ -532,6 +535,8 @@ class YoutubeDL(object):
    def save_console_title(self):
        if not self.params.get('consoletitle', False):
            return
        if self.params.get('simulate', False):
            return
        if compat_os_name != 'nt' and 'TERM' in os.environ:
            # Save the title on stack
            self._write_string('\033[22;0t', self._screen_file)
@ -539,6 +544,8 @@ class YoutubeDL(object):
    def restore_console_title(self):
        if not self.params.get('consoletitle', False):
            return
        if self.params.get('simulate', False):
            return
        if compat_os_name != 'nt' and 'TERM' in os.environ:
            # Restore the title from stack
            self._write_string('\033[23;0t', self._screen_file)
@ -1475,23 +1482,28 @@ class YoutubeDL(object):
            if info_dict.get('%s_number' % field) is not None and not info_dict.get(field):
                info_dict[field] = '%s %d' % (field.capitalize(), info_dict['%s_number' % field])
        for cc_kind in ('subtitles', 'automatic_captions'):
            cc = info_dict.get(cc_kind)
            if cc:
                for _, subtitle in cc.items():
                    for subtitle_format in subtitle:
                        if subtitle_format.get('url'):
                            subtitle_format['url'] = sanitize_url(subtitle_format['url'])
                        if subtitle_format.get('ext') is None:
                            subtitle_format['ext'] = determine_ext(subtitle_format['url']).lower()
        automatic_captions = info_dict.get('automatic_captions')
        subtitles = info_dict.get('subtitles')
        if subtitles:
            for _, subtitle in subtitles.items():
                for subtitle_format in subtitle:
                    if subtitle_format.get('url'):
                        subtitle_format['url'] = sanitize_url(subtitle_format['url'])
                    if subtitle_format.get('ext') is None:
                        subtitle_format['ext'] = determine_ext(subtitle_format['url']).lower()
        if self.params.get('listsubtitles', False):
            if 'automatic_captions' in info_dict:
-                self.list_subtitles(info_dict['id'], info_dict.get('automatic_captions'), 'automatic captions')
+                self.list_subtitles(
                    info_dict['id'], automatic_captions, 'automatic captions')
            self.list_subtitles(info_dict['id'], subtitles, 'subtitles')
            return
        info_dict['requested_subtitles'] = self.process_subtitles(
-            info_dict['id'], subtitles,
+            info_dict['id'], subtitles, automatic_captions)
            info_dict.get('automatic_captions'))
        # We now pick which formats have to be downloaded
        if info_dict.get('formats') is None:
@ -1849,7 +1861,7 @@ class YoutubeDL(object):
                    def compatible_formats(formats):
                        video, audio = formats
                        # Check extension
-                        video_ext, audio_ext = audio.get('ext'), video.get('ext')
+                        video_ext, audio_ext = video.get('ext'), audio.get('ext')
                        if video_ext and audio_ext:
                            COMPATIBLE_EXTS = (
                                ('mp3', 'mp4', 'm4a', 'm4p', 'm4b', 'm4r', 'm4v', 'ismv', 'isma'),
--- a/youtube_dl/init.py
+++ b/youtube_dl/init.py
@ -430,6 +430,7 @@ def _real_main(argv=None):
        'config_location': opts.config_location,
        'geo_bypass': opts.geo_bypass,
        'geo_bypass_country': opts.geo_bypass_country,
        'geo_bypass_ip_block': opts.geo_bypass_ip_block,
        # just for deprecation check
        'autonumber': opts.autonumber if opts.autonumber is True else None,
        'usetitle': opts.usetitle if opts.usetitle is True else None,
--- a/youtube_dl/compat.py
+++ b/youtube_dl/compat.py
@ -2787,6 +2787,12 @@ except NameError:  # Python 3
    compat_numeric_types = (int, float, complex)
 try:
    compat_integer_types = (int, long)
 except NameError:  # Python 3
    compat_integer_types = (int, )
 if sys.version_info < (2, 7):
    def compat_socket_create_connection(address, timeout, source_address=None):
        host, port = address
@ -2974,6 +2980,7 @@ __all__ = [
    'compat_http_client',
    'compat_http_server',
    'compat_input',
    'compat_integer_types',
    'compat_itertools_count',
    'compat_kwargs',
    'compat_numeric_types',
--- a/youtube_dl/downloader/common.py
+++ b/youtube_dl/downloader/common.py
@ -45,7 +45,6 @@ class FileDownloader(object):
    min_filesize:       Skip files smaller than this size
    max_filesize:       Skip files larger than this size
    xattr_set_filesize: Set ytdl.filesize user xattribute with expected size.
                        (experimental)
    external_downloader_args:  A list of additional command-line arguments for the
                        external downloader.
    hls_use_mpegts:     Use the mpegts container for HLS videos.
--- a/youtube_dl/downloader/dash.py
+++ b/youtube_dl/downloader/dash.py
@ -2,7 +2,10 @@ from __future__ import unicode_literals
 from .fragment import FragmentFD
 from ..compat import compat_urllib_error
-from ..utils import urljoin
+from ..utils import (
    DownloadError,
    urljoin,
 )
 class DashSegmentsFD(FragmentFD):
@ -57,6 +60,14 @@ class DashSegmentsFD(FragmentFD):
                    count += 1
                    if count <= fragment_retries:
                        self.report_retry_fragment(err, frag_index, count, fragment_retries)
                except DownloadError:
                    # Don't retry fragment if error occurred during HTTP downloading
                    # itself since it has own retry settings
                    if not fatal:
                        self.report_skip_fragment(frag_index)
                        break
                    raise
            if count > fragment_retries:
                if not fatal:
                    self.report_skip_fragment(frag_index)
--- a/youtube_dl/downloader/fragment.py
+++ b/youtube_dl/downloader/fragment.py
@ -74,9 +74,14 @@ class FragmentFD(FileDownloader):
        return not ctx['live'] and not ctx['tmpfilename'] == '-'
    def _read_ytdl_file(self, ctx):
        assert 'ytdl_corrupt' not in ctx
        stream, _ = sanitize_open(self.ytdl_filename(ctx['filename']), 'r')
-        ctx['fragment_index'] = json.loads(stream.read())['downloader']['current_fragment']['index']
+        try:
-        stream.close()
+            ctx['fragment_index'] = json.loads(stream.read())['downloader']['current_fragment']['index']
        except Exception:
            ctx['ytdl_corrupt'] = True
        finally:
            stream.close()
    def _write_ytdl_file(self, ctx):
        frag_index_stream, _ = sanitize_open(self.ytdl_filename(ctx['filename']), 'w')
@ -158,11 +163,17 @@ class FragmentFD(FileDownloader):
        if self.__do_ytdl_file(ctx):
            if os.path.isfile(encodeFilename(self.ytdl_filename(ctx['filename']))):
                self._read_ytdl_file(ctx)
-                if ctx['fragment_index'] > 0 and resume_len == 0:
+                is_corrupt = ctx.get('ytdl_corrupt') is True
                is_inconsistent = ctx['fragment_index'] > 0 and resume_len == 0
                if is_corrupt or is_inconsistent:
                    message = (
                        '.ytdl file is corrupt' if is_corrupt else
                        'Inconsistent state of incomplete fragment download')
                    self.report_warning(
-                        'Inconsistent state of incomplete fragment download. '
+                        '%s. Restarting from the beginning...' % message)
                        'Restarting from the beginning...')
                    ctx['fragment_index'] = resume_len = 0
                    if 'ytdl_corrupt' in ctx:
                        del ctx['ytdl_corrupt']
                    self._write_ytdl_file(ctx)
            else:
                self._write_ytdl_file(ctx)
--- a/youtube_dl/downloader/http.py
+++ b/youtube_dl/downloader/http.py
@ -217,10 +217,11 @@ class HttpFD(FileDownloader):
            before = start  # start measuring
            def retry(e):
-                if ctx.tmpfilename != '-':
+                to_stdout = ctx.tmpfilename == '-'
                if not to_stdout:
                    ctx.stream.close()
                ctx.stream = None
-                ctx.resume_len = os.path.getsize(encodeFilename(ctx.tmpfilename))
+                ctx.resume_len = byte_counter if to_stdout else os.path.getsize(encodeFilename(ctx.tmpfilename))
                raise RetryDownload(e)
            while True:
--- a/youtube_dl/downloader/rtmp.py
+++ b/youtube_dl/downloader/rtmp.py
@ -29,66 +29,68 @@ class RtmpFD(FileDownloader):
            proc = subprocess.Popen(args, stderr=subprocess.PIPE)
            cursor_in_new_line = True
            proc_stderr_closed = False
-            while not proc_stderr_closed:
+            try:
-                # read line from stderr
+                while not proc_stderr_closed:
-                line = ''
+                    # read line from stderr
-                while True:
+                    line = ''
-                    char = proc.stderr.read(1)
+                    while True:
-                    if not char:
+                        char = proc.stderr.read(1)
-                        proc_stderr_closed = True
+                        if not char:
-                        break
+                            proc_stderr_closed = True
-                    if char in [b'\r', b'\n']:
+                            break
-                        break
+                        if char in [b'\r', b'\n']:
-                    line += char.decode('ascii', 'replace')
+                            break
-                if not line:
+                        line += char.decode('ascii', 'replace')
-                    # proc_stderr_closed is True
+                    if not line:
-                    continue
+                        # proc_stderr_closed is True
-                mobj = re.search(r'([0-9]+\.[0-9]{3}) kB / [0-9]+\.[0-9]{2} sec \(([0-9]{1,2}\.[0-9])%\)', line)
+                        continue
-                if mobj:
+                    mobj = re.search(r'([0-9]+\.[0-9]{3}) kB / [0-9]+\.[0-9]{2} sec \(([0-9]{1,2}\.[0-9])%\)', line)
                    downloaded_data_len = int(float(mobj.group(1)) * 1024)
                    percent = float(mobj.group(2))
                    if not resume_percent:
                        resume_percent = percent
                        resume_downloaded_data_len = downloaded_data_len
                    time_now = time.time()
                    eta = self.calc_eta(start, time_now, 100 - resume_percent, percent - resume_percent)
                    speed = self.calc_speed(start, time_now, downloaded_data_len - resume_downloaded_data_len)
                    data_len = None
                    if percent > 0:
                        data_len = int(downloaded_data_len * 100 / percent)
                    self._hook_progress({
                        'status': 'downloading',
                        'downloaded_bytes': downloaded_data_len,
                        'total_bytes_estimate': data_len,
                        'tmpfilename': tmpfilename,
                        'filename': filename,
                        'eta': eta,
                        'elapsed': time_now - start,
                        'speed': speed,
                    })
                    cursor_in_new_line = False
                else:
                    # no percent for live streams
                    mobj = re.search(r'([0-9]+\.[0-9]{3}) kB / [0-9]+\.[0-9]{2} sec', line)
                    if mobj:
                        downloaded_data_len = int(float(mobj.group(1)) * 1024)
                        percent = float(mobj.group(2))
                        if not resume_percent:
                            resume_percent = percent
                            resume_downloaded_data_len = downloaded_data_len
                        time_now = time.time()
-                        speed = self.calc_speed(start, time_now, downloaded_data_len)
+                        eta = self.calc_eta(start, time_now, 100 - resume_percent, percent - resume_percent)
                        speed = self.calc_speed(start, time_now, downloaded_data_len - resume_downloaded_data_len)
                        data_len = None
                        if percent > 0:
                            data_len = int(downloaded_data_len * 100 / percent)
                        self._hook_progress({
                            'status': 'downloading',
                            'downloaded_bytes': downloaded_data_len,
                            'total_bytes_estimate': data_len,
                            'tmpfilename': tmpfilename,
                            'filename': filename,
-                            'status': 'downloading',
+                            'eta': eta,
                            'elapsed': time_now - start,
                            'speed': speed,
                        })
                        cursor_in_new_line = False
-                    elif self.params.get('verbose', False):
+                    else:
-                        if not cursor_in_new_line:
+                        # no percent for live streams
-                            self.to_screen('')
+                        mobj = re.search(r'([0-9]+\.[0-9]{3}) kB / [0-9]+\.[0-9]{2} sec', line)
-                        cursor_in_new_line = True
+                        if mobj:
-                        self.to_screen('[rtmpdump] ' + line)
+                            downloaded_data_len = int(float(mobj.group(1)) * 1024)
-            proc.wait()
+                            time_now = time.time()
                            speed = self.calc_speed(start, time_now, downloaded_data_len)
                            self._hook_progress({
                                'downloaded_bytes': downloaded_data_len,
                                'tmpfilename': tmpfilename,
                                'filename': filename,
                                'status': 'downloading',
                                'elapsed': time_now - start,
                                'speed': speed,
                            })
                            cursor_in_new_line = False
                        elif self.params.get('verbose', False):
                            if not cursor_in_new_line:
                                self.to_screen('')
                            cursor_in_new_line = True
                            self.to_screen('[rtmpdump] ' + line)
            finally:
                proc.wait()
            if not cursor_in_new_line:
                self.to_screen('')
            return proc.returncode
@ -163,7 +165,15 @@ class RtmpFD(FileDownloader):
        RD_INCOMPLETE = 2
        RD_NO_CONNECT = 3
-        retval = run_rtmpdump(args)
+        started = time.time()
        try:
            retval = run_rtmpdump(args)
        except KeyboardInterrupt:
            if not info_dict.get('is_live'):
                raise
            retval = RD_SUCCESS
            self.to_screen('\n[rtmpdump] Interrupted by user')
        if retval == RD_NO_CONNECT:
            self.report_error('[rtmpdump] Could not connect to RTMP server.')
@ -171,7 +181,7 @@ class RtmpFD(FileDownloader):
        while retval in (RD_INCOMPLETE, RD_FAILED) and not test and not live:
            prevsize = os.path.getsize(encodeFilename(tmpfilename))
-            self.to_screen('[rtmpdump] %s bytes' % prevsize)
+            self.to_screen('[rtmpdump] Downloaded %s bytes' % prevsize)
            time.sleep(5.0)  # This seems to be needed
            args = basic_args + ['--resume']
            if retval == RD_FAILED:
@ -188,13 +198,14 @@ class RtmpFD(FileDownloader):
                break
        if retval == RD_SUCCESS or (test and retval == RD_INCOMPLETE):
            fsize = os.path.getsize(encodeFilename(tmpfilename))
-            self.to_screen('[rtmpdump] %s bytes' % fsize)
+            self.to_screen('[rtmpdump] Downloaded %s bytes' % fsize)
            self.try_rename(tmpfilename, filename)
            self._hook_progress({
                'downloaded_bytes': fsize,
                'total_bytes': fsize,
                'filename': filename,
                'status': 'finished',
                'elapsed': time.time() - started,
            })
            return True
        else:
--- a/youtube_dl/extractor/abc.py
+++ b/youtube_dl/extractor/abc.py
@ -13,6 +13,7 @@ from ..utils import (
    int_or_none,
    parse_iso8601,
    try_get,
    unescapeHTML,
    update_url_query,
 )
@ -104,21 +105,22 @@ class ABCIE(InfoExtractor):
 class ABCIViewIE(InfoExtractor):
    IE_NAME = 'abc.net.au:iview'
-    _VALID_URL = r'https?://iview\.abc\.net\.au/programs/[^/]+/(?P<id>[^/?#]+)'
+    _VALID_URL = r'https?://iview\.abc\.net\.au/(?:[^/]+/)*video/(?P<id>[^/?#]+)'
    _GEO_COUNTRIES = ['AU']
    # ABC iview programs are normally available for 14 days only.
    _TESTS = [{
-        'url': 'http://iview.abc.net.au/programs/call-the-midwife/ZW0898A003S00',
+        'url': 'https://iview.abc.net.au/show/ben-and-hollys-little-kingdom/series/0/video/ZX9371A050S00',
        'md5': 'cde42d728b3b7c2b32b1b94b4a548afc',
        'info_dict': {
-            'id': 'ZW0898A003S00',
+            'id': 'ZX9371A050S00',
            'ext': 'mp4',
-            'title': 'Series 5 Ep 3',
+            'title': "Gaston's Birthday",
-            'description': 'md5:e0ef7d4f92055b86c4f33611f180ed79',
+            'series': "Ben And Holly's Little Kingdom",
-            'upload_date': '20171228',
+            'description': 'md5:f9de914d02f226968f598ac76f105bcf',
-            'uploader_id': 'abc1',
+            'upload_date': '20180604',
-            'timestamp': 1514499187,
+            'uploader_id': 'abc4kids',
            'timestamp': 1528140219,
        },
        'params': {
            'skip_download': True,
@ -127,17 +129,16 @@ class ABCIViewIE(InfoExtractor):
    def _real_extract(self, url):
        video_id = self._match_id(url)
-        webpage = self._download_webpage(url, video_id)
+        video_params = self._download_json(
-        video_params = self._parse_json(self._search_regex(
+            'https://iview.abc.net.au/api/programs/' + video_id, video_id)
-            r'videoParams\s*=\s*({.+?});', webpage, 'video params'), video_id)
+        title = unescapeHTML(video_params.get('title') or video_params['seriesTitle'])
-        title = video_params.get('title') or video_params['seriesTitle']
+        stream = next(s for s in video_params['playlist'] if s.get('type') in ('program', 'livestream'))
        stream = next(s for s in video_params['playlist'] if s.get('type') == 'program')
-        house_number = video_params.get('episodeHouseNumber')
+        house_number = video_params.get('episodeHouseNumber') or video_id
-        path = '/auth/hls/sign?ts={0}&hn={1}&d=android-mobile'.format(
+        path = '/auth/hls/sign?ts={0}&hn={1}&d=android-tablet'.format(
            int(time.time()), house_number)
        sig = hmac.new(
-            'android.content.res.Resources'.encode('utf-8'),
+            b'android.content.res.Resources',
            path.encode('utf-8'), hashlib.sha256).hexdigest()
        token = self._download_webpage(
            'http://iview.abc.net.au{0}&sig={1}'.format(path, sig), video_id)
@ -167,18 +168,26 @@ class ABCIViewIE(InfoExtractor):
                'ext': 'vtt',
            }]
        is_live = video_params.get('livestream') == '1'
        if is_live:
            title = self._live_title(title)
        return {
            'id': video_id,
            'title': title,
-            'description': self._html_search_meta(['og:description', 'twitter:description'], webpage),
+            'description': video_params.get('description'),
-            'thumbnail': self._html_search_meta(['og:image', 'twitter:image:src'], webpage),
+            'thumbnail': video_params.get('thumbnail'),
            'duration': int_or_none(video_params.get('eventDuration')),
            'timestamp': parse_iso8601(video_params.get('pubDate'), ' '),
-            'series': video_params.get('seriesTitle'),
+            'series': unescapeHTML(video_params.get('seriesTitle')),
            'series_id': video_params.get('seriesHouseNumber') or video_id[:7],
-            'episode_number': int_or_none(self._html_search_meta('episodeNumber', webpage, default=None)),
+            'season_number': int_or_none(self._search_regex(
-            'episode': self._html_search_meta('episode_title', webpage, default=None),
+                r'\bSeries\s+(\d+)\b', title, 'season number', default=None)),
            'episode_number': int_or_none(self._search_regex(
                r'\bEp\s+(\d+)\b', title, 'episode number', default=None)),
            'episode_id': house_number,
            'uploader_id': video_params.get('channel'),
            'formats': formats,
            'subtitles': subtitles,
            'is_live': is_live,
        }
--- a/youtube_dl/extractor/acast.py
+++ b/youtube_dl/extractor/acast.py
@ -7,7 +7,9 @@ import functools
 from .common import InfoExtractor
 from ..compat import compat_str
 from ..utils import (
    float_or_none,
    int_or_none,
    try_get,
    unified_timestamp,
    OnDemandPagedList,
 )
@ -24,40 +26,58 @@ class ACastIE(InfoExtractor):
            'id': '57de3baa-4bb0-487e-9418-2692c1277a34',
            'ext': 'mp3',
            'title': '"Where Are You?": Taipei 101, Taiwan',
            'description': 'md5:a0b4ef3634e63866b542e5b1199a1a0e',
            'timestamp': 1196172000,
            'upload_date': '20071127',
            'description': 'md5:a0b4ef3634e63866b542e5b1199a1a0e',
            'duration': 211,
            'creator': 'Concierge',
            'series': 'Condé Nast Traveler Podcast',
            'episode': '"Where Are You?": Taipei 101, Taiwan',
        }
    }, {
        # test with multiple blings
        'url': 'https://www.acast.com/sparpodcast/2.raggarmordet-rosterurdetforflutna',
-        'md5': 'e87d5b8516cd04c0d81b6ee1caca28d0',
+        'md5': 'a02393c74f3bdb1801c3ec2695577ce0',
        'info_dict': {
            'id': '2a92b283-1a75-4ad8-8396-499c641de0d9',
            'ext': 'mp3',
            'title': '2. Raggarmordet - Röster ur det förflutna',
            'description': 'md5:4f81f6d8cf2e12ee21a321d8bca32db4',
            'timestamp': 1477346700,
            'upload_date': '20161024',
-            'description': 'md5:4f81f6d8cf2e12ee21a321d8bca32db4',
+            'duration': 2766.602563,
-            'duration': 2766,
+            'creator': 'Anton Berg & Martin Johnson',
            'series': 'Spår',
            'episode': '2. Raggarmordet - Röster ur det förflutna',
        }
    }]
    def _real_extract(self, url):
        channel, display_id = re.match(self._VALID_URL, url).groups()
        s = self._download_json(
            'https://play-api.acast.com/stitch/%s/%s' % (channel, display_id),
            display_id)['result']
        media_url = s['url']
        cast_data = self._download_json(
-            'https://play-api.acast.com/splash/%s/%s' % (channel, display_id), display_id)
+            'https://play-api.acast.com/splash/%s/%s' % (channel, display_id),
-        e = cast_data['result']['episode']
+            display_id)['result']
        e = cast_data['episode']
        title = e['name']
        return {
            'id': compat_str(e['id']),
            'display_id': display_id,
-            'url': e['mediaUrl'],
+            'url': media_url,
-            'title': e['name'],
+            'title': title,
-            'description': e.get('description'),
+            'description': e.get('description') or e.get('summary'),
            'thumbnail': e.get('image'),
            'timestamp': unified_timestamp(e.get('publishingDate')),
-            'duration': int_or_none(e.get('duration')),
+            'duration': float_or_none(s.get('duration') or e.get('duration')),
            'filesize': int_or_none(e.get('contentLength')),
            'creator': try_get(cast_data, lambda x: x['show']['author'], compat_str),
            'series': try_get(cast_data, lambda x: x['show']['name'], compat_str),
            'season_number': int_or_none(e.get('seasonNumber')),
            'episode': title,
            'episode_number': int_or_none(e.get('episodeNumber')),
        }
--- a/youtube_dl/extractor/adn.py
+++ b/youtube_dl/extractor/adn.py
@ -1,8 +1,11 @@
 # coding: utf-8
 from __future__ import unicode_literals
 import base64
 import binascii
 import json
 import os
 import random
 from .common import InfoExtractor
 from ..aes import aes_cbc_decrypt
@ -12,9 +15,12 @@ from ..compat import (
 )
 from ..utils import (
    bytes_to_intlist,
    bytes_to_long,
    ExtractorError,
    float_or_none,
    intlist_to_bytes,
    long_to_bytes,
    pkcs1pad,
    srt_subtitles_timecode,
    strip_or_none,
    urljoin,
@ -35,6 +41,7 @@ class ADNIE(InfoExtractor):
        }
    }
    _BASE_URL = 'http://animedigitalnetwork.fr'
    _RSA_KEY = (0xc35ae1e4356b65a73b551493da94b8cb443491c0aa092a357a5aee57ffc14dda85326f42d716e539a34542a0d3f363adf16c5ec222d713d5997194030ee2e4f0d1fb328c01a81cf6868c090d50de8e169c6b13d1675b9eeed1cbc51e1fffca9b38af07f37abd790924cd3bee59d0257cfda4fe5f3f0534877e21ce5821447d1b, 65537)
    def _get_subtitles(self, sub_path, video_id):
        if not sub_path:
@ -42,16 +49,14 @@ class ADNIE(InfoExtractor):
        enc_subtitles = self._download_webpage(
            urljoin(self._BASE_URL, sub_path),
-            video_id, fatal=False, headers={
+            video_id, fatal=False)
                'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:53.0) Gecko/20100101 Firefox/53.0',
            })
        if not enc_subtitles:
            return None
        # http://animedigitalnetwork.fr/components/com_vodvideo/videojs/adn-vjs.min.js
        dec_subtitles = intlist_to_bytes(aes_cbc_decrypt(
            bytes_to_intlist(compat_b64decode(enc_subtitles[24:])),
-            bytes_to_intlist(b'\xc8\x6e\x06\xbc\xbe\xc6\x49\xf5\x88\x0d\xc8\x47\xc4\x27\x0c\x60'),
+            bytes_to_intlist(binascii.unhexlify(self._K + '9032ad7083106400')),
            bytes_to_intlist(compat_b64decode(enc_subtitles[:24]))
        ))
        subtitles_json = self._parse_json(
@ -112,11 +117,24 @@ class ADNIE(InfoExtractor):
        error = None
        if not links:
            links_url = player_config.get('linksurl') or options['videoUrl']
-            links_data = self._download_json(urljoin(
+            token = options['token']
-                self._BASE_URL, links_url), video_id)
+            self._K = ''.join([random.choice('0123456789abcdef') for _ in range(16)])
            message = bytes_to_intlist(json.dumps({
                'k': self._K,
                'e': 60,
                't': token,
            }))
            padded_message = intlist_to_bytes(pkcs1pad(message, 128))
            n, e = self._RSA_KEY
            encrypted_message = long_to_bytes(pow(bytes_to_long(padded_message), e, n))
            authorization = base64.b64encode(encrypted_message).decode()
            links_data = self._download_json(
                urljoin(self._BASE_URL, links_url), video_id, headers={
                    'Authorization': 'Bearer ' + authorization,
                })
            links = links_data.get('links') or {}
            metas = metas or links_data.get('meta') or {}
-            sub_path = sub_path or links_data.get('subtitles')
+            sub_path = (sub_path or links_data.get('subtitles')) + '&token=' + token
            error = links_data.get('error')
        title = metas.get('title') or video_info['title']
--- a/youtube_dl/extractor/adobepass.py
+++ b/youtube_dl/extractor/adobepass.py
@ -1325,8 +1325,8 @@ class AdobePassIE(InfoExtractor):
    _DOWNLOADING_LOGIN_PAGE = 'Downloading Provider Login Page'
    def _download_webpage_handle(self, *args, **kwargs):
-        headers = kwargs.get('headers', {})
+        headers = self.geo_verification_headers()
-        headers.update(self.geo_verification_headers())
+        headers.update(kwargs.get('headers', {}))
        kwargs['headers'] = headers
        return super(AdobePassIE, self)._download_webpage_handle(
            *args, **compat_kwargs(kwargs))
--- a/youtube_dl/extractor/adultswim.py
+++ b/youtube_dl/extractor/adultswim.py
@ -7,6 +7,7 @@ from .turner import TurnerBaseIE
 from ..utils import (
    int_or_none,
    strip_or_none,
    url_or_none,
 )
@ -98,7 +99,7 @@ class AdultSwimIE(TurnerBaseIE):
            if not video_id:
                entries = []
                for episode in video_data.get('archiveEpisodes', []):
-                    episode_url = episode.get('url')
+                    episode_url = url_or_none(episode.get('url'))
                    if not episode_url:
                        continue
                    entries.append(self.url_result(
--- a/youtube_dl/extractor/afreecatv.py
+++ b/youtube_dl/extractor/afreecatv.py
@ -9,6 +9,8 @@ from ..utils import (
    determine_ext,
    ExtractorError,
    int_or_none,
    url_or_none,
    urlencode_postdata,
    xpath_text,
 )
@ -28,6 +30,7 @@ class AfreecaTVIE(InfoExtractor):
                        )
                        (?P<id>\d+)
                    '''
    _NETRC_MACHINE = 'afreecatv'
    _TESTS = [{
        'url': 'http://live.afreecatv.com:8079/app/index.cgi?szType=read_ucc_bbs&szBjId=dailyapril&nStationNo=16711924&nBbsNo=18605867&nTitleNo=36164052&szSkin=',
        'md5': 'f72c89fe7ecc14c1b5ce506c4996046e',
@ -139,22 +142,22 @@ class AfreecaTVIE(InfoExtractor):
            'skip_download': True,
        },
    }, {
-        # adult video
+        # PARTIAL_ADULT
-        'url': 'http://vod.afreecatv.com/PLAYER/STATION/26542731',
+        'url': 'http://vod.afreecatv.com/PLAYER/STATION/32028439',
        'info_dict': {
-            'id': '20171001_F1AE1711_196617479_1',
+            'id': '20180327_27901457_202289533_1',
            'ext': 'mp4',
-            'title': '[생]서아 초심 찾기 방송 (part 1)',
+            'title': '[생]빨개요♥ (part 1)',
            'thumbnail': 're:^https?://(?:video|st)img.afreecatv.com/.*$',
-            'uploader': 'BJ서아',
+            'uploader': '[SA]서아',
            'uploader_id': 'bjdyrksu',
-            'upload_date': '20171001',
+            'upload_date': '20180327',
-            'duration': 3600,
+            'duration': 3601,
            'age_limit': 18,
        },
        'params': {
            'skip_download': True,
        },
        'expected_warnings': ['adult content'],
    }, {
        'url': 'http://www.afreecatv.com/player/Player.swf?szType=szBjId=djleegoon&nStationNo=11273158&nBbsNo=13161095&nTitleNo=36327652',
        'only_matching': True,
@ -172,6 +175,51 @@ class AfreecaTVIE(InfoExtractor):
            video_key['part'] = int(m.group('part'))
        return video_key
    def _real_initialize(self):
        self._login()
    def _login(self):
        username, password = self._get_login_info()
        if username is None:
            return
        login_form = {
            'szWork': 'login',
            'szType': 'json',
            'szUid': username,
            'szPassword': password,
            'isSaveId': 'false',
            'szScriptVar': 'oLoginRet',
            'szAction': '',
        }
        response = self._download_json(
            'https://login.afreecatv.com/app/LoginAction.php', None,
            'Logging in', data=urlencode_postdata(login_form))
        _ERRORS = {
            -4: 'Your account has been suspended due to a violation of our terms and policies.',
            -5: 'https://member.afreecatv.com/app/user_delete_progress.php',
            -6: 'https://login.afreecatv.com/membership/changeMember.php',
            -8: "Hello! AfreecaTV here.\nThe username you have entered belongs to \n an account that requires a legal guardian's consent. \nIf you wish to use our services without restriction, \nplease make sure to go through the necessary verification process.",
            -9: 'https://member.afreecatv.com/app/pop_login_block.php',
            -11: 'https://login.afreecatv.com/afreeca/second_login.php',
            -12: 'https://member.afreecatv.com/app/user_security.php',
            0: 'The username does not exist or you have entered the wrong password.',
            -1: 'The username does not exist or you have entered the wrong password.',
            -3: 'You have entered your username/password incorrectly.',
            -7: 'You cannot use your Global AfreecaTV account to access Korean AfreecaTV.',
            -10: 'Sorry for the inconvenience. \nYour account has been blocked due to an unauthorized access. \nPlease contact our Help Center for assistance.',
            -32008: 'You have failed to log in. Please contact our Help Center.',
        }
        result = int_or_none(response.get('RESULT'))
        if result != 1:
            error = _ERRORS.get(result, 'You have failed to log in.')
            raise ExtractorError(
                'Unable to login: %s said: %s' % (self.IE_NAME, error),
                expected=True)
    def _real_extract(self, url):
        video_id = self._match_id(url)
@ -187,22 +235,42 @@ class AfreecaTVIE(InfoExtractor):
            r'nBbsNo\s*=\s*(\d+)', webpage, 'bbs')
        video_id = self._search_regex(
            r'nTitleNo\s*=\s*(\d+)', webpage, 'title', default=video_id)
-        print(video_id, station_id, bbs_id)
+
-        video_xml = self._download_xml(
+        partial_view = False
-            'http://afbbs.afreecatv.com:8080/api/video/get_video_info.php',
+        for _ in range(2):
-            video_id, headers={
+            query = {
                'Referer': url,
            }, query={
                'nTitleNo': video_id,
                'nStationNo': station_id,
                'nBbsNo': bbs_id,
-                'partialView': 'SKIP_ADULT',
+            }
-            })
+            if partial_view:
                query['partialView'] = 'SKIP_ADULT'
            video_xml = self._download_xml(
                'http://afbbs.afreecatv.com:8080/api/video/get_video_info.php',
                video_id, 'Downloading video info XML%s'
                % (' (skipping adult)' if partial_view else ''),
                video_id, headers={
                    'Referer': url,
                }, query=query)
-        flag = xpath_text(video_xml, './track/flag', 'flag', default=None)
+            flag = xpath_text(video_xml, './track/flag', 'flag', default=None)
-        if flag and flag != 'SUCCEED':
+            if flag and flag == 'SUCCEED':
                break
            if flag == 'PARTIAL_ADULT':
                self._downloader.report_warning(
                    'In accordance with local laws and regulations, underage users are restricted from watching adult content. '
                    'Only content suitable for all ages will be downloaded. '
                    'Provide account credentials if you wish to download restricted content.')
                partial_view = True
                continue
            elif flag == 'ADULT':
                error = 'Only users older than 19 are able to watch this video. Provide account credentials to download this content.'
            else:
                error = flag
            raise ExtractorError(
-                '%s said: %s' % (self.IE_NAME, flag), expected=True)
+                '%s said: %s' % (self.IE_NAME, error), expected=True)
        else:
            raise ExtractorError('Unable to download video info')
        video_element = video_xml.findall(compat_xpath('./track/video'))[-1]
        if video_element is None or video_element.text is None:
@ -237,7 +305,7 @@ class AfreecaTVIE(InfoExtractor):
            file_elements = video_element.findall(compat_xpath('./file'))
            one = len(file_elements) == 1
            for file_num, file_element in enumerate(file_elements, start=1):
-                file_url = file_element.text
+                file_url = url_or_none(file_element.text)
                if not file_url:
                    continue
                key = file_element.get('key', '')
--- a/youtube_dl/extractor/americastestkitchen.py
+++ b/youtube_dl/extractor/americastestkitchen.py
--- a/youtube_dl/extractor/amp.py
+++ b/youtube_dl/extractor/amp.py
@ -3,11 +3,12 @@ from __future__ import unicode_literals
 from .common import InfoExtractor
 from ..utils import (
    int_or_none,
    parse_iso8601,
    mimetype2ext,
    determine_ext,
    ExtractorError,
    int_or_none,
    mimetype2ext,
    parse_iso8601,
    url_or_none,
 )
@ -35,7 +36,7 @@ class AMPIE(InfoExtractor):
                media_thumbnail = [media_thumbnail]
            for thumbnail_data in media_thumbnail:
                thumbnail = thumbnail_data.get('@attributes', {})
-                thumbnail_url = thumbnail.get('url')
+                thumbnail_url = url_or_none(thumbnail.get('url'))
                if not thumbnail_url:
                    continue
                thumbnails.append({
@ -51,7 +52,7 @@ class AMPIE(InfoExtractor):
                media_subtitle = [media_subtitle]
            for subtitle_data in media_subtitle:
                subtitle = subtitle_data.get('@attributes', {})
-                subtitle_href = subtitle.get('href')
+                subtitle_href = url_or_none(subtitle.get('href'))
                if not subtitle_href:
                    continue
                subtitles.setdefault(subtitle.get('lang') or 'en', []).append({
@ -65,7 +66,7 @@ class AMPIE(InfoExtractor):
            media_content = [media_content]
        for media_data in media_content:
            media = media_data.get('@attributes', {})
-            media_url = media.get('url')
+            media_url = url_or_none(media.get('url'))
            if not media_url:
                continue
            ext = mimetype2ext(media.get('type')) or determine_ext(media_url)
@ -79,7 +80,7 @@ class AMPIE(InfoExtractor):
            else:
                formats.append({
                    'format_id': media_data.get('media-category', {}).get('@attributes', {}).get('label'),
-                    'url': media['url'],
+                    'url': media_url,
                    'tbr': int_or_none(media.get('bitrate')),
                    'filesize': int_or_none(media.get('fileSize')),
                    'ext': ext,
--- a/youtube_dl/extractor/animeondemand.py
+++ b/youtube_dl/extractor/animeondemand.py
@ -8,6 +8,7 @@ from ..utils import (
    determine_ext,
    extract_attributes,
    ExtractorError,
    url_or_none,
    urlencode_postdata,
    urljoin,
 )
@ -52,7 +53,7 @@ class AnimeOnDemandIE(InfoExtractor):
    }]
    def _login(self):
-        (username, password) = self._get_login_info()
+        username, password = self._get_login_info()
        if username is None:
            return
@ -165,7 +166,7 @@ class AnimeOnDemandIE(InfoExtractor):
                        }, fatal=False)
                    if not playlist:
                        continue
-                    stream_url = playlist.get('streamurl')
+                    stream_url = url_or_none(playlist.get('streamurl'))
                    if stream_url:
                        rtmp = re.search(
                            r'^(?P<url>rtmpe?://(?P<host>[^/]+)/(?P<app>.+/))(?P<playpath>mp[34]:.+)',
--- a/youtube_dl/extractor/anvato.py
+++ b/youtube_dl/extractor/anvato.py
@ -134,9 +134,33 @@ class AnvatoIE(InfoExtractor):
        'telemundo': 'anvato_mcp_telemundo_web_prod_c5278d51ad46fda4b6ca3d0ea44a7846a054f582'
    }
    _API_KEY = '3hwbSuqqT690uxjNYBktSQpa5ZrpYYR0Iofx7NcJHyA'
    _ANVP_RE = r'<script[^>]+\bdata-anvp\s*=\s*(["\'])(?P<anvp>(?:(?!\1).)+)\1'
    _AUTH_KEY = b'\x31\xc2\x42\x84\x9e\x73\xa0\xce'
    _TESTS = [{
        # from https://www.boston25news.com/news/watch-humpback-whale-breaches-right-next-to-fishing-boat-near-nh/817484874
        'url': 'anvato:8v9BEynrwx8EFLYpgfOWcG1qJqyXKlRM:4465496',
        'info_dict': {
            'id': '4465496',
            'ext': 'mp4',
            'title': 'VIDEO: Humpback whale breaches right next to NH boat',
            'description': 'VIDEO: Humpback whale breaches right next to NH boat. Footage courtesy: Zach Fahey.',
            'duration': 22,
            'timestamp': 1534855680,
            'upload_date': '20180821',
            'uploader': 'ANV',
        },
        'params': {
            'skip_download': True,
        },
    }, {
        # from https://sanfrancisco.cbslocal.com/2016/06/17/source-oakland-cop-on-leave-for-having-girlfriend-help-with-police-reports/
        'url': 'anvato:DVzl9QRzox3ZZsP9bNu5Li3X7obQOnqP:3417601',
        'only_matching': True,
    }]
    def __init__(self, *args, **kwargs):
        super(AnvatoIE, self).__init__(*args, **kwargs)
        self.__server_time = None
@ -169,7 +193,8 @@ class AnvatoIE(InfoExtractor):
            'api': {
                'anvrid': anvrid,
                'anvstk': md5_text('%s|%s|%d|%s' % (
-                    access_key, anvrid, server_time, self._ANVACK_TABLE[access_key])),
+                    access_key, anvrid, server_time,
                    self._ANVACK_TABLE.get(access_key, self._API_KEY))),
                'anvts': server_time,
            },
        }
@ -277,10 +302,13 @@ class AnvatoIE(InfoExtractor):
    def _real_extract(self, url):
        url, smuggled_data = unsmuggle_url(url, {})
-        self._initialize_geo_bypass(smuggled_data.get('geo_countries'))
+        self._initialize_geo_bypass({
            'countries': smuggled_data.get('geo_countries'),
        })
        mobj = re.match(self._VALID_URL, url)
        access_key, video_id = mobj.group('access_key_or_mcp', 'id')
        if access_key not in self._ANVACK_TABLE:
-            access_key = self._MCP_TO_ACCESS_KEY_TABLE[access_key]
+            access_key = self._MCP_TO_ACCESS_KEY_TABLE.get(
                access_key) or access_key
        return self._get_anvato_videos(access_key, video_id)
--- a/youtube_dl/extractor/aol.py
+++ b/youtube_dl/extractor/aol.py
@ -7,6 +7,7 @@ from .common import InfoExtractor
 from ..utils import (
    ExtractorError,
    int_or_none,
    url_or_none,
 )
@ -77,7 +78,7 @@ class AolIE(InfoExtractor):
            formats.extend(self._extract_m3u8_formats(
                m3u8_url, video_id, 'mp4', m3u8_id='hls', fatal=False))
        for rendition in video_data.get('renditions', []):
-            video_url = rendition.get('url')
+            video_url = url_or_none(rendition.get('url'))
            if not video_url:
                continue
            ext = rendition.get('format')
--- a/youtube_dl/extractor/apa.py
+++ b/youtube_dl/extractor/apa.py
@ -0,0 +1,94 @@
 # coding: utf-8
 from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
 from ..utils import (
    determine_ext,
    js_to_json,
    url_or_none,
 )
 class APAIE(InfoExtractor):
    _VALID_URL = r'https?://[^/]+\.apa\.at/embed/(?P<id>[\da-f]{8}-[\da-f]{4}-[\da-f]{4}-[\da-f]{4}-[\da-f]{12})'
    _TESTS = [{
        'url': 'http://uvp.apa.at/embed/293f6d17-692a-44e3-9fd5-7b178f3a1029',
        'md5': '2b12292faeb0a7d930c778c7a5b4759b',
        'info_dict': {
            'id': 'jjv85FdZ',
            'ext': 'mp4',
            'title': '"Blau ist mysteriös": Die Blue Man Group im Interview',
            'description': 'md5:d41d8cd98f00b204e9800998ecf8427e',
            'thumbnail': r're:^https?://.*\.jpg$',
            'duration': 254,
            'timestamp': 1519211149,
            'upload_date': '20180221',
        },
    }, {
        'url': 'https://uvp-apapublisher.sf.apa.at/embed/2f94e9e6-d945-4db2-9548-f9a41ebf7b78',
        'only_matching': True,
    }, {
        'url': 'http://uvp-rma.sf.apa.at/embed/70404cca-2f47-4855-bbb8-20b1fae58f76',
        'only_matching': True,
    }, {
        'url': 'http://uvp-kleinezeitung.sf.apa.at/embed/f1c44979-dba2-4ebf-b021-e4cf2cac3c81',
        'only_matching': True,
    }]
    @staticmethod
    def _extract_urls(webpage):
        return [
            mobj.group('url')
            for mobj in re.finditer(
                r'<iframe[^>]+\bsrc=(["\'])(?P<url>(?:https?:)?//[^/]+\.apa\.at/embed/[\da-f]{8}-[\da-f]{4}-[\da-f]{4}-[\da-f]{4}-[\da-f]{12}.*?)\1',
                webpage)]
    def _real_extract(self, url):
        video_id = self._match_id(url)
        webpage = self._download_webpage(url, video_id)
        jwplatform_id = self._search_regex(
            r'media[iI]d\s*:\s*["\'](?P<id>[a-zA-Z0-9]{8})', webpage,
            'jwplatform id', default=None)
        if jwplatform_id:
            return self.url_result(
                'jwplatform:' + jwplatform_id, ie='JWPlatform',
                video_id=video_id)
        sources = self._parse_json(
            self._search_regex(
                r'sources\s*=\s*(\[.+?\])\s*;', webpage, 'sources'),
            video_id, transform_source=js_to_json)
        formats = []
        for source in sources:
            if not isinstance(source, dict):
                continue
            source_url = url_or_none(source.get('file'))
            if not source_url:
                continue
            ext = determine_ext(source_url)
            if ext == 'm3u8':
                formats.extend(self._extract_m3u8_formats(
                    source_url, video_id, 'mp4', entry_protocol='m3u8_native',
                    m3u8_id='hls', fatal=False))
            else:
                formats.append({
                    'url': source_url,
                })
        self._sort_formats(formats)
        thumbnail = self._search_regex(
            r'image\s*:\s*(["\'])(?P<url>(?:(?!\1).)+)\1', webpage,
            'thumbnail', fatal=False, group='url')
        return {
            'id': video_id,
            'title': video_id,
            'thumbnail': thumbnail,
            'formats': formats,
        }
--- a/youtube_dl/extractor/aparat.py
+++ b/youtube_dl/extractor/aparat.py
@ -5,6 +5,7 @@ from .common import InfoExtractor
 from ..utils import (
    int_or_none,
    mimetype2ext,
    url_or_none,
 )
@ -43,7 +44,7 @@ class AparatIE(InfoExtractor):
        formats = []
        for item in file_list[0]:
-            file_url = item.get('file')
+            file_url = url_or_none(item.get('file'))
            if not file_url:
                continue
            ext = mimetype2ext(item.get('type'))
--- a/youtube_dl/extractor/ard.py
+++ b/youtube_dl/extractor/ard.py
@ -5,7 +5,6 @@ import re
 from .common import InfoExtractor
 from .generic import GenericIE
 from ..compat import compat_str
 from ..utils import (
    determine_ext,
    ExtractorError,
@ -15,13 +14,14 @@ from ..utils import (
    unified_strdate,
    xpath_text,
    update_url_query,
    url_or_none,
 )
 from ..compat import compat_etree_fromstring
 class ARDMediathekIE(InfoExtractor):
    IE_NAME = 'ARD:mediathek'
-    _VALID_URL = r'^https?://(?:(?:www\.)?ardmediathek\.de|mediathek\.(?:daserste|rbb-online)\.de)/(?:.*/)(?P<video_id>[0-9]+|[^0-9][^/\?]+)[^/\?]*(?:\?.*)?'
+    _VALID_URL = r'^https?://(?:(?:www\.)?ardmediathek\.de|mediathek\.(?:daserste|rbb-online)\.de|one\.ard\.de)/(?:.*/)(?P<video_id>[0-9]+|[^0-9][^/\?]+)[^/\?]*(?:\?.*)?'
    _TESTS = [{
        # available till 26.07.2022
@ -37,6 +37,9 @@ class ARDMediathekIE(InfoExtractor):
            # m3u8 download
            'skip_download': True,
        }
    }, {
        'url': 'https://one.ard.de/tv/Mord-mit-Aussicht/Mord-mit-Aussicht-6-39-T%C3%B6dliche-Nach/ONE/Video?bcastId=46384294&documentId=55586872',
        'only_matching': True,
    }, {
        # audio
        'url': 'http://www.ardmediathek.de/tv/WDR-H%C3%B6rspiel-Speicher/Tod-eines-Fu%C3%9Fballers/WDR-3/Audio-Podcast?documentId=28488308&bcastId=23074086',
@ -100,7 +103,7 @@ class ARDMediathekIE(InfoExtractor):
                quality = stream.get('_quality')
                server = stream.get('_server')
                for stream_url in stream_urls:
-                    if not isinstance(stream_url, compat_str) or '//' not in stream_url:
+                    if not url_or_none(stream_url):
                        continue
                    ext = determine_ext(stream_url)
                    if quality != 'auto' and ext in ('f4m', 'm3u8'):
@ -282,3 +285,76 @@ class ARDIE(InfoExtractor):
            'upload_date': upload_date,
            'thumbnail': thumbnail,
        }
 class ARDBetaMediathekIE(InfoExtractor):
    _VALID_URL = r'https://beta\.ardmediathek\.de/[a-z]+/player/(?P<video_id>[a-zA-Z0-9]+)/(?P<display_id>[^/?#]+)'
    _TESTS = [{
        'url': 'https://beta.ardmediathek.de/ard/player/Y3JpZDovL2Rhc2Vyc3RlLmRlL3RhdG9ydC9mYmM4NGM1NC0xNzU4LTRmZGYtYWFhZS0wYzcyZTIxNGEyMDE/die-robuste-roswita',
        'md5': '2d02d996156ea3c397cfc5036b5d7f8f',
        'info_dict': {
            'display_id': 'die-robuste-roswita',
            'id': 'Y3JpZDovL2Rhc2Vyc3RlLmRlL3RhdG9ydC9mYmM4NGM1NC0xNzU4LTRmZGYtYWFhZS0wYzcyZTIxNGEyMDE',
            'title': 'Tatort: Die robuste Roswita',
            'description': r're:^Der Mord.*trüber ist als die Ilm.',
            'duration': 5316,
            'thumbnail': 'https://img.ardmediathek.de/standard/00/55/43/59/34/-1774185891/16x9/960?mandant=ard',
            'upload_date': '20180826',
            'ext': 'mp4',
        },
    }]
    def _real_extract(self, url):
        mobj = re.match(self._VALID_URL, url)
        video_id = mobj.group('video_id')
        display_id = mobj.group('display_id')
        webpage = self._download_webpage(url, display_id)
        data_json = self._search_regex(r'window\.__APOLLO_STATE__\s*=\s*(\{.*);\n', webpage, 'json')
        data = self._parse_json(data_json, display_id)
        res = {
            'id': video_id,
            'display_id': display_id,
        }
        formats = []
        for widget in data.values():
            if widget.get('_geoblocked'):
                raise ExtractorError('This video is not available due to geoblocking', expected=True)
            if '_duration' in widget:
                res['duration'] = widget['_duration']
            if 'clipTitle' in widget:
                res['title'] = widget['clipTitle']
            if '_previewImage' in widget:
                res['thumbnail'] = widget['_previewImage']
            if 'broadcastedOn' in widget:
                res['upload_date'] = unified_strdate(widget['broadcastedOn'])
            if 'synopsis' in widget:
                res['description'] = widget['synopsis']
            if '_subtitleUrl' in widget:
                res['subtitles'] = {'de': [{
                    'ext': 'ttml',
                    'url': widget['_subtitleUrl'],
                }]}
            if '_quality' in widget:
                format_url = widget['_stream']['json'][0]
                if format_url.endswith('.f4m'):
                    formats.extend(self._extract_f4m_formats(
                        format_url + '?hdcore=3.11.0',
                        video_id, f4m_id='hds', fatal=False))
                elif format_url.endswith('m3u8'):
                    formats.extend(self._extract_m3u8_formats(
                        format_url, video_id, 'mp4', m3u8_id='hls', fatal=False))
                else:
                    formats.append({
                        'format_id': 'http-' + widget['_quality'],
                        'url': format_url,
                        'preference': 10,  # Plain HTTP, that's nice
                    })
        self._sort_formats(formats)
        res['formats'] = formats
        return res
--- a/youtube_dl/extractor/asiancrush.py
+++ b/youtube_dl/extractor/asiancrush.py
@ -8,7 +8,6 @@ from .kaltura import KalturaIE
 from ..utils import (
    extract_attributes,
    remove_end,
    urlencode_postdata,
 )
@ -34,19 +33,40 @@ class AsianCrushIE(InfoExtractor):
    def _real_extract(self, url):
        video_id = self._match_id(url)
-        data = self._download_json(
+        webpage = self._download_webpage(url, video_id)
            'https://www.asiancrush.com/wp-admin/admin-ajax.php', video_id,
            data=urlencode_postdata({
                'postid': video_id,
                'action': 'get_channel_kaltura_vars',
            }))
-        entry_id = data['entry_id']
+        entry_id, partner_id, title = [None] * 3
        vars = self._parse_json(
            self._search_regex(
                r'iEmbedVars\s*=\s*({.+?})', webpage, 'embed vars',
                default='{}'), video_id, fatal=False)
        if vars:
            entry_id = vars.get('entry_id')
            partner_id = vars.get('partner_id')
            title = vars.get('vid_label')
        if not entry_id:
            entry_id = self._search_regex(
                r'\bentry_id["\']\s*:\s*["\'](\d+)', webpage, 'entry id')
        player = self._download_webpage(
            'https://api.asiancrush.com/embeddedVideoPlayer', video_id,
            query={'id': entry_id})
        kaltura_id = self._search_regex(
            r'entry_id["\']\s*:\s*(["\'])(?P<id>(?:(?!\1).)+)\1', player,
            'kaltura id', group='id')
        if not partner_id:
            partner_id = self._search_regex(
                r'/p(?:artner_id)?/(\d+)', player, 'partner id',
                default='513551')
        return self.url_result(
-            'kaltura:%s:%s' % (data['partner_id'], entry_id),
+            'kaltura:%s:%s' % (partner_id, kaltura_id),
-            ie=KalturaIE.ie_key(), video_id=entry_id,
+            ie=KalturaIE.ie_key(), video_id=kaltura_id,
-            video_title=data.get('vid_label'))
+            video_title=title)
 class AsianCrushPlaylistIE(InfoExtractor):
--- a/youtube_dl/extractor/atresplayer.py
+++ b/youtube_dl/extractor/atresplayer.py
@ -74,7 +74,7 @@ class AtresPlayerIE(InfoExtractor):
        self._login()
    def _login(self):
-        (username, password) = self._get_login_info()
+        username, password = self._get_login_info()
        if username is None:
            return
--- a/youtube_dl/extractor/audimedia.py
+++ b/youtube_dl/extractor/audimedia.py
@ -5,13 +5,12 @@ from .common import InfoExtractor
 from ..utils import (
    int_or_none,
    parse_iso8601,
    sanitized_Request,
 )
 class AudiMediaIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?audi-mediacenter\.com/(?:en|de)/audimediatv/(?P<id>[^/?#]+)'
+    _VALID_URL = r'https?://(?:www\.)?audi-mediacenter\.com/(?:en|de)/audimediatv/(?:video/)?(?P<id>[^/?#]+)'
-    _TEST = {
+    _TESTS = [{
        'url': 'https://www.audi-mediacenter.com/en/audimediatv/60-seconds-of-audi-sport-104-2015-wec-bahrain-rookie-test-1467',
        'md5': '79a8b71c46d49042609795ab59779b66',
        'info_dict': {
@ -24,41 +23,46 @@ class AudiMediaIE(InfoExtractor):
            'duration': 74022,
            'view_count': int,
        }
-    }
+    }, {
-    # extracted from https://audimedia.tv/assets/embed/embedded-player.js (dataSourceAuthToken)
+        'url': 'https://www.audi-mediacenter.com/en/audimediatv/video/60-seconds-of-audi-sport-104-2015-wec-bahrain-rookie-test-2991',
-    _AUTH_TOKEN = 'e25b42847dba18c6c8816d5d8ce94c326e06823ebf0859ed164b3ba169be97f2'
+        'only_matching': True,
    }]
    def _real_extract(self, url):
        display_id = self._match_id(url)
        webpage = self._download_webpage(url, display_id)
        raw_payload = self._search_regex([
-            r'class="amtv-embed"[^>]+id="([^"]+)"',
+            r'class="amtv-embed"[^>]+id="([0-9a-z-]+)"',
-            r'class=\\"amtv-embed\\"[^>]+id=\\"([^"]+)\\"',
+            r'id="([0-9a-z-]+)"[^>]+class="amtv-embed"',
            r'class=\\"amtv-embed\\"[^>]+id=\\"([0-9a-z-]+)\\"',
            r'id=\\"([0-9a-z-]+)\\"[^>]+class=\\"amtv-embed\\"',
            r'id=(?:\\)?"(amtve-[a-z]-\d+-[a-z]{2})',
        ], webpage, 'raw payload')
-        _, stage_mode, video_id, lang = raw_payload.split('-')
+        _, stage_mode, video_id, _ = raw_payload.split('-')
        # TODO: handle s and e stage_mode (live streams and ended live streams)
        if stage_mode not in ('s', 'e'):
-            request = sanitized_Request(
+            video_data = self._download_json(
-                'https://audimedia.tv/api/video/v1/videos/%s?embed[]=video_versions&embed[]=thumbnail_image&where[content_language_iso]=%s' % (video_id, lang),
+                'https://www.audimedia.tv/api/video/v1/videos/' + video_id,
-                headers={'X-Auth-Token': self._AUTH_TOKEN})
+                video_id, query={
-            json_data = self._download_json(request, video_id)['results']
+                    'embed[]': ['video_versions', 'thumbnail_image'],
                })['results']
            formats = []
-            stream_url_hls = json_data.get('stream_url_hls')
+            stream_url_hls = video_data.get('stream_url_hls')
            if stream_url_hls:
                formats.extend(self._extract_m3u8_formats(
                    stream_url_hls, video_id, 'mp4',
                    entry_protocol='m3u8_native', m3u8_id='hls', fatal=False))
-            stream_url_hds = json_data.get('stream_url_hds')
+            stream_url_hds = video_data.get('stream_url_hds')
            if stream_url_hds:
                formats.extend(self._extract_f4m_formats(
                    stream_url_hds + '?hdcore=3.4.0',
                    video_id, f4m_id='hds', fatal=False))
-            for video_version in json_data.get('video_versions'):
+            for video_version in video_data.get('video_versions', []):
                video_version_url = video_version.get('download_url') or video_version.get('stream_url')
                if not video_version_url:
                    continue
@ -79,11 +83,11 @@ class AudiMediaIE(InfoExtractor):
            return {
                'id': video_id,
-                'title': json_data['title'],
+                'title': video_data['title'],
-                'description': json_data.get('subtitle'),
+                'description': video_data.get('subtitle'),
-                'thumbnail': json_data.get('thumbnail_image', {}).get('file'),
+                'thumbnail': video_data.get('thumbnail_image', {}).get('file'),
-                'timestamp': parse_iso8601(json_data.get('publication_date')),
+                'timestamp': parse_iso8601(video_data.get('publication_date')),
-                'duration': int_or_none(json_data.get('duration')),
+                'duration': int_or_none(video_data.get('duration')),
-                'view_count': int_or_none(json_data.get('view_count')),
+                'view_count': int_or_none(video_data.get('view_count')),
                'formats': formats,
            }
--- a/youtube_dl/extractor/audiomack.py
+++ b/youtube_dl/extractor/audiomack.py
@ -65,7 +65,7 @@ class AudiomackIE(InfoExtractor):
            return {'_type': 'url', 'url': api_response['url'], 'ie_key': 'Soundcloud'}
        return {
-            'id': api_response.get('id', album_url_tag),
+            'id': compat_str(api_response.get('id', album_url_tag)),
            'uploader': api_response.get('artist'),
            'title': api_response.get('title'),
            'url': api_response['url'],
--- a/youtube_dl/extractor/bambuser.py
+++ b/youtube_dl/extractor/bambuser.py
@ -44,7 +44,7 @@ class BambuserIE(InfoExtractor):
    }
    def _login(self):
-        (username, password) = self._get_login_info()
+        username, password = self._get_login_info()
        if username is None:
            return
--- a/youtube_dl/extractor/bandcamp.py
+++ b/youtube_dl/extractor/bandcamp.py
@ -1,6 +1,5 @@
 from __future__ import unicode_literals
 import json
 import random
 import re
 import time
@ -16,14 +15,18 @@ from ..utils import (
    int_or_none,
    KNOWN_EXTENSIONS,
    parse_filesize,
    str_or_none,
    try_get,
    unescapeHTML,
    update_url_query,
    unified_strdate,
    unified_timestamp,
    url_or_none,
 )
 class BandcampIE(InfoExtractor):
-    _VALID_URL = r'https?://.*?\.bandcamp\.com/track/(?P<title>[^/?#&]+)'
+    _VALID_URL = r'https?://[^/]+\.bandcamp\.com/track/(?P<title>[^/?#&]+)'
    _TESTS = [{
        'url': 'http://youtube-dl.bandcamp.com/track/youtube-dl-test-song',
        'md5': 'c557841d5e50261777a6585648adf439',
@ -35,13 +38,44 @@ class BandcampIE(InfoExtractor):
        },
        '_skip': 'There is a limit of 200 free downloads / month for the test song'
    }, {
        # free download
        'url': 'http://benprunty.bandcamp.com/track/lanius-battle',
-        'md5': '0369ace6b939f0927e62c67a1a8d9fa7',
+        'md5': '853e35bf34aa1d6fe2615ae612564b36',
        'info_dict': {
            'id': '2650410135',
            'ext': 'aiff',
            'title': 'Ben Prunty - Lanius (Battle)',
            'thumbnail': r're:^https?://.*\.jpg$',
            'uploader': 'Ben Prunty',
            'timestamp': 1396508491,
            'upload_date': '20140403',
            'release_date': '20140403',
            'duration': 260.877,
            'track': 'Lanius (Battle)',
            'track_number': 1,
            'track_id': '2650410135',
            'artist': 'Ben Prunty',
            'album': 'FTL: Advanced Edition Soundtrack',
        },
    }, {
        # no free download, mp3 128
        'url': 'https://relapsealumni.bandcamp.com/track/hail-to-fire',
        'md5': 'fec12ff55e804bb7f7ebeb77a800c8b7',
        'info_dict': {
            'id': '2584466013',
            'ext': 'mp3',
            'title': 'Mastodon - Hail to Fire',
            'thumbnail': r're:^https?://.*\.jpg$',
            'uploader': 'Mastodon',
            'timestamp': 1322005399,
            'upload_date': '20111122',
            'release_date': '20040207',
            'duration': 120.79,
            'track': 'Hail to Fire',
            'track_number': 5,
            'track_id': '2584466013',
            'artist': 'Mastodon',
            'album': 'Call of the Mastodon',
        },
    }]
@ -50,19 +84,23 @@ class BandcampIE(InfoExtractor):
        title = mobj.group('title')
        webpage = self._download_webpage(url, title)
        thumbnail = self._html_search_meta('og:image', webpage, default=None)
        m_download = re.search(r'freeDownloadPage: "(.*?)"', webpage)
        if not m_download:
            m_trackinfo = re.search(r'trackinfo: (.+),\s*?\n', webpage)
            if m_trackinfo:
                json_code = m_trackinfo.group(1)
                data = json.loads(json_code)[0]
                track_id = compat_str(data['id'])
-                if not data.get('file'):
+        track_id = None
-                    raise ExtractorError('Not streamable', video_id=track_id, expected=True)
+        track = None
        track_number = None
        duration = None
-                formats = []
+        formats = []
-                for format_id, format_url in data['file'].items():
+        track_info = self._parse_json(
            self._search_regex(
                r'trackinfo\s*:\s*\[\s*({.+?})\s*\]\s*,\s*?\n',
                webpage, 'track info', default='{}'), title)
        if track_info:
            file_ = track_info.get('file')
            if isinstance(file_, dict):
                for format_id, format_url in file_.items():
                    if not url_or_none(format_url):
                        continue
                    ext, abr_str = format_id.split('-', 1)
                    formats.append({
                        'format_id': format_id,
@ -72,85 +110,110 @@ class BandcampIE(InfoExtractor):
                        'acodec': ext,
                        'abr': int_or_none(abr_str),
                    })
            track = track_info.get('title')
            track_id = str_or_none(track_info.get('track_id') or track_info.get('id'))
            track_number = int_or_none(track_info.get('track_num'))
            duration = float_or_none(track_info.get('duration'))
-                self._sort_formats(formats)
+        def extract(key):
            return self._search_regex(
                r'\b%s\s*["\']?\s*:\s*(["\'])(?P<value>(?:(?!\1).)+)\1' % key,
                webpage, key, default=None, group='value')
-                return {
+        artist = extract('artist')
-                    'id': track_id,
+        album = extract('album_title')
-                    'title': data['title'],
+        timestamp = unified_timestamp(
-                    'thumbnail': thumbnail,
+            extract('publish_date') or extract('album_publish_date'))
-                    'formats': formats,
+        release_date = unified_strdate(extract('album_release_date'))
                    'duration': float_or_none(data.get('duration')),
                }
            else:
                raise ExtractorError('No free songs found')
-        download_link = m_download.group(1)
+        download_link = self._search_regex(
-        video_id = self._search_regex(
+            r'freeDownloadPage\s*:\s*(["\'])(?P<url>(?:(?!\1).)+)\1', webpage,
-            r'(?ms)var TralbumData = .*?[{,]\s*id: (?P<id>\d+),?$',
+            'download link', default=None, group='url')
-            webpage, 'video id')
+        if download_link:
            track_id = self._search_regex(
                r'(?ms)var TralbumData = .*?[{,]\s*id: (?P<id>\d+),?$',
                webpage, 'track id')
-        download_webpage = self._download_webpage(
+            download_webpage = self._download_webpage(
-            download_link, video_id, 'Downloading free downloads page')
+                download_link, track_id, 'Downloading free downloads page')
-        blob = self._parse_json(
+            blob = self._parse_json(
-            self._search_regex(
+                self._search_regex(
-                r'data-blob=(["\'])(?P<blob>{.+?})\1', download_webpage,
+                    r'data-blob=(["\'])(?P<blob>{.+?})\1', download_webpage,
-                'blob', group='blob'),
+                    'blob', group='blob'),
-            video_id, transform_source=unescapeHTML)
+                track_id, transform_source=unescapeHTML)
-        info = blob['digital_items'][0]
+            info = try_get(
                blob, (lambda x: x['digital_items'][0],
                       lambda x: x['download_items'][0]), dict)
            if info:
                downloads = info.get('downloads')
                if isinstance(downloads, dict):
                    if not track:
                        track = info.get('title')
                    if not artist:
                        artist = info.get('artist')
                    if not thumbnail:
                        thumbnail = info.get('thumb_url')
-        downloads = info['downloads']
+                    download_formats = {}
-        track = info['title']
+                    download_formats_list = blob.get('download_formats')
                    if isinstance(download_formats_list, list):
                        for f in blob['download_formats']:
                            name, ext = f.get('name'), f.get('file_extension')
                            if all(isinstance(x, compat_str) for x in (name, ext)):
                                download_formats[name] = ext.strip('.')
-        artist = info.get('artist')
+                    for format_id, f in downloads.items():
-        title = '%s - %s' % (artist, track) if artist else track
+                        format_url = f.get('url')
                        if not format_url:
                            continue
                        # Stat URL generation algorithm is reverse engineered from
                        # download_*_bundle_*.js
                        stat_url = update_url_query(
                            format_url.replace('/download/', '/statdownload/'), {
                                '.rand': int(time.time() * 1000 * random.random()),
                            })
                        format_id = f.get('encoding_name') or format_id
                        stat = self._download_json(
                            stat_url, track_id, 'Downloading %s JSON' % format_id,
                            transform_source=lambda s: s[s.index('{'):s.rindex('}') + 1],
                            fatal=False)
                        if not stat:
                            continue
                        retry_url = url_or_none(stat.get('retry_url'))
                        if not retry_url:
                            continue
                        formats.append({
                            'url': self._proto_relative_url(retry_url, 'http:'),
                            'ext': download_formats.get(format_id),
                            'format_id': format_id,
                            'format_note': f.get('description'),
                            'filesize': parse_filesize(f.get('size_mb')),
                            'vcodec': 'none',
                        })
        download_formats = {}
        for f in blob['download_formats']:
            name, ext = f.get('name'), f.get('file_extension')
            if all(isinstance(x, compat_str) for x in (name, ext)):
                download_formats[name] = ext.strip('.')
        formats = []
        for format_id, f in downloads.items():
            format_url = f.get('url')
            if not format_url:
                continue
            # Stat URL generation algorithm is reverse engineered from
            # download_*_bundle_*.js
            stat_url = update_url_query(
                format_url.replace('/download/', '/statdownload/'), {
                    '.rand': int(time.time() * 1000 * random.random()),
                })
            format_id = f.get('encoding_name') or format_id
            stat = self._download_json(
                stat_url, video_id, 'Downloading %s JSON' % format_id,
                transform_source=lambda s: s[s.index('{'):s.rindex('}') + 1],
                fatal=False)
            if not stat:
                continue
            retry_url = stat.get('retry_url')
            if not isinstance(retry_url, compat_str):
                continue
            formats.append({
                'url': self._proto_relative_url(retry_url, 'http:'),
                'ext': download_formats.get(format_id),
                'format_id': format_id,
                'format_note': f.get('description'),
                'filesize': parse_filesize(f.get('size_mb')),
                'vcodec': 'none',
            })
        self._sort_formats(formats)
        title = '%s - %s' % (artist, track) if artist else track
        if not duration:
            duration = float_or_none(self._html_search_meta(
                'duration', webpage, default=None))
        return {
-            'id': video_id,
+            'id': track_id,
            'title': title,
-            'thumbnail': info.get('thumb_url') or thumbnail,
+            'thumbnail': thumbnail,
-            'uploader': info.get('artist'),
+            'uploader': artist,
-            'artist': artist,
+            'timestamp': timestamp,
            'release_date': release_date,
            'duration': duration,
            'track': track,
            'track_number': track_number,
            'track_id': track_id,
            'artist': artist,
            'album': album,
            'formats': formats,
        }
@ -306,7 +369,7 @@ class BandcampWeeklyIE(InfoExtractor):
        formats = []
        for format_id, format_url in show['audio_stream'].items():
-            if not isinstance(format_url, compat_str):
+            if not url_or_none(format_url):
                continue
            for known_ext in KNOWN_EXTENSIONS:
                if known_ext in format_id:
--- a/youtube_dl/extractor/bbc.py
+++ b/youtube_dl/extractor/bbc.py
@ -12,6 +12,7 @@ from ..utils import (
    float_or_none,
    get_element_by_class,
    int_or_none,
    js_to_json,
    parse_duration,
    parse_iso8601,
    try_get,
@ -20,7 +21,6 @@ from ..utils import (
    urljoin,
 )
 from ..compat import (
    compat_etree_fromstring,
    compat_HTTPError,
    compat_urlparse,
 )
@ -29,7 +29,7 @@ from ..compat import (
 class BBCCoUkIE(InfoExtractor):
    IE_NAME = 'bbc.co.uk'
    IE_DESC = 'BBC iPlayer'
-    _ID_REGEX = r'[pbw][\da-z]{7}'
+    _ID_REGEX = r'(?:[pbm][\da-z]{7}|w[\da-z]{7,14})'
    _VALID_URL = r'''(?x)
                    https?://
                        (?:www\.)?bbc\.co\.uk/
@ -236,6 +236,12 @@ class BBCCoUkIE(InfoExtractor):
        }, {
            'url': 'http://www.bbc.co.uk/programmes/w3csv1y9',
            'only_matching': True,
        }, {
            'url': 'https://www.bbc.co.uk/programmes/m00005xn',
            'only_matching': True,
        }, {
            'url': 'https://www.bbc.co.uk/programmes/w172w4dww1jqt5s',
            'only_matching': True,
        }]
    _USP_RE = r'/([^/]+?)\.ism(?:\.hlsv2\.ism)?/[^/]+\.m3u8'
@ -333,14 +339,9 @@ class BBCCoUkIE(InfoExtractor):
        self._raise_extractor_error(last_exception)
    def _download_media_selector_url(self, url, programme_id=None):
-        try:
+        media_selection = self._download_xml(
-            media_selection = self._download_xml(
+            url, programme_id, 'Downloading media selection XML',
-                url, programme_id, 'Downloading media selection XML')
+            expected_status=(403, 404))
        except ExtractorError as ee:
            if isinstance(ee.cause, compat_HTTPError) and ee.cause.code in (403, 404):
                media_selection = compat_etree_fromstring(ee.cause.read().decode('utf-8'))
            else:
                raise
        return self._process_media_selector(media_selection, programme_id)
    def _process_media_selector(self, media_selection, programme_id):
@ -772,6 +773,28 @@ class BBCIE(BBCCoUkIE):
        # single video article embedded with data-media-vpid
        'url': 'http://www.bbc.co.uk/sport/rowing/35908187',
        'only_matching': True,
    }, {
        'url': 'https://www.bbc.co.uk/bbcthree/clip/73d0bbd0-abc3-4cea-b3c0-cdae21905eb1',
        'info_dict': {
            'id': 'p06556y7',
            'ext': 'mp4',
            'title': 'Transfers: Cristiano Ronaldo to Man Utd, Arsenal to spend?',
            'description': 'md5:4b7dfd063d5a789a1512e99662be3ddd',
        },
        'params': {
            'skip_download': True,
        }
    }, {
        # window.__PRELOADED_STATE__
        'url': 'https://www.bbc.co.uk/radio/play/b0b9z4yl',
        'info_dict': {
            'id': 'b0b9z4vz',
            'ext': 'mp4',
            'title': 'Prom 6: An American in Paris and Turangalila',
            'description': 'md5:51cf7d6f5c8553f197e58203bc78dff8',
            'uploader': 'Radio 3',
            'uploader_id': 'bbc_radio_three',
        },
    }]
    @classmethod
@ -994,6 +1017,66 @@ class BBCIE(BBCCoUkIE):
                    'subtitles': subtitles,
                }
        preload_state = self._parse_json(self._search_regex(
            r'window\.__PRELOADED_STATE__\s*=\s*({.+?});', webpage,
            'preload state', default='{}'), playlist_id, fatal=False)
        if preload_state:
            current_programme = preload_state.get('programmes', {}).get('current') or {}
            programme_id = current_programme.get('id')
            if current_programme and programme_id and current_programme.get('type') == 'playable_item':
                title = current_programme.get('titles', {}).get('tertiary') or playlist_title
                formats, subtitles = self._download_media_selector(programme_id)
                self._sort_formats(formats)
                synopses = current_programme.get('synopses') or {}
                network = current_programme.get('network') or {}
                duration = int_or_none(
                    current_programme.get('duration', {}).get('value'))
                thumbnail = None
                image_url = current_programme.get('image_url')
                if image_url:
                    thumbnail = image_url.replace('{recipe}', '1920x1920')
                return {
                    'id': programme_id,
                    'title': title,
                    'description': dict_get(synopses, ('long', 'medium', 'short')),
                    'thumbnail': thumbnail,
                    'duration': duration,
                    'uploader': network.get('short_title'),
                    'uploader_id': network.get('id'),
                    'formats': formats,
                    'subtitles': subtitles,
                }
        bbc3_config = self._parse_json(
            self._search_regex(
                r'(?s)bbcthreeConfig\s*=\s*({.+?})\s*;\s*<', webpage,
                'bbcthree config', default='{}'),
            playlist_id, transform_source=js_to_json, fatal=False)
        if bbc3_config:
            bbc3_playlist = try_get(
                bbc3_config, lambda x: x['payload']['content']['bbcMedia']['playlist'],
                dict)
            if bbc3_playlist:
                playlist_title = bbc3_playlist.get('title') or playlist_title
                thumbnail = bbc3_playlist.get('holdingImageURL')
                entries = []
                for bbc3_item in bbc3_playlist['items']:
                    programme_id = bbc3_item.get('versionID')
                    if not programme_id:
                        continue
                    formats, subtitles = self._download_media_selector(programme_id)
                    self._sort_formats(formats)
                    entries.append({
                        'id': programme_id,
                        'title': playlist_title,
                        'thumbnail': thumbnail,
                        'timestamp': timestamp,
                        'formats': formats,
                        'subtitles': subtitles,
                    })
                return self.playlist_result(
                    entries, playlist_id, playlist_title, playlist_description)
        def extract_all(pattern):
            return list(filter(None, map(
                lambda s: self._parse_json(s, playlist_id, fatal=False),
--- a/youtube_dl/extractor/bellmedia.py
+++ b/youtube_dl/extractor/bellmedia.py
@ -12,7 +12,7 @@ class BellMediaIE(InfoExtractor):
            (?:
                ctv|
                tsn|
-                bnn|
+                bnn(?:bloomberg)?|
                thecomedynetwork|
                discovery|
                discoveryvelocity|
@ -27,17 +27,16 @@ class BellMediaIE(InfoExtractor):
            much\.com
        )/.*?(?:\bvid(?:eoid)?=|-vid|~|%7E|/(?:episode)?)(?P<id>[0-9]{6,})'''
    _TESTS = [{
-        'url': 'http://www.ctv.ca/video/player?vid=706966',
+        'url': 'https://www.bnnbloomberg.ca/video/david-cockfield-s-top-picks~1403070',
-        'md5': 'ff2ebbeae0aa2dcc32a830c3fd69b7b0',
+        'md5': '36d3ef559cfe8af8efe15922cd3ce950',
        'info_dict': {
-            'id': '706966',
+            'id': '1403070',
-            'ext': 'mp4',
+            'ext': 'flv',
-            'title': 'Larry Day and Richard Jutras on the TIFF red carpet of \'Stonewall\'',
+            'title': 'David Cockfield\'s Top Picks',
-            'description': 'etalk catches up with Larry Day and Richard Jutras on the TIFF red carpet of "Stonewall”.',
+            'description': 'md5:810f7f8c6a83ad5b48677c3f8e5bb2c3',
-            'upload_date': '20150919',
+            'upload_date': '20180525',
-            'timestamp': 1442624700,
+            'timestamp': 1527288600,
        },
        'expected_warnings': ['HTTP Error 404'],
    }, {
        'url': 'http://www.thecomedynetwork.ca/video/player?vid=923582',
        'only_matching': True,
@ -70,6 +69,7 @@ class BellMediaIE(InfoExtractor):
        'investigationdiscovery': 'invdisc',
        'animalplanet': 'aniplan',
        'etalk': 'ctv',
        'bnnbloomberg': 'bnn',
    }
    def _real_extract(self, url):
--- a/youtube_dl/extractor/bilibili.py
+++ b/youtube_dl/extractor/bilibili.py
@ -114,12 +114,12 @@ class BiliBiliIE(InfoExtractor):
        if 'anime/' not in url:
            cid = self._search_regex(
-                r'cid(?:["\']:|=)(\d+)', webpage, 'cid',
+                r'\bcid(?:["\']:|=)(\d+)', webpage, 'cid',
                default=None
            ) or compat_parse_qs(self._search_regex(
-                [r'1EmbedPlayer\([^)]+,\s*"([^"]+)"\)',
+                [r'EmbedPlayer\([^)]+,\s*"([^"]+)"\)',
-                 r'1EmbedPlayer\([^)]+,\s*\\"([^"]+)\\"\)',
+                 r'EmbedPlayer\([^)]+,\s*\\"([^"]+)\\"\)',
-                 r'1<iframe[^>]+src="https://secure\.bilibili\.com/secure,([^"]+)"'],
+                 r'<iframe[^>]+src="https://secure\.bilibili\.com/secure,([^"]+)"'],
                webpage, 'player parameters'))['cid'][0]
        else:
            if 'no_bangumi_tip' not in smuggled_data:
--- a/youtube_dl/extractor/bitchute.py
+++ b/youtube_dl/extractor/bitchute.py
@ -0,0 +1,120 @@
 # coding: utf-8
 from __future__ import unicode_literals
 import itertools
 import re
 from .common import InfoExtractor
 from ..utils import urlencode_postdata
 class BitChuteIE(InfoExtractor):
    _VALID_URL = r'https?://(?:www\.)?bitchute\.com/(?:video|embed|torrent/[^/]+)/(?P<id>[^/?#&]+)'
    _TESTS = [{
        'url': 'https://www.bitchute.com/video/szoMrox2JEI/',
        'md5': '66c4a70e6bfc40dcb6be3eb1d74939eb',
        'info_dict': {
            'id': 'szoMrox2JEI',
            'ext': 'mp4',
            'title': 'Fuck bitches get money',
            'description': 'md5:3f21f6fb5b1d17c3dee9cf6b5fe60b3a',
            'thumbnail': r're:^https?://.*\.jpg$',
            'uploader': 'Victoria X Rave',
        },
    }, {
        'url': 'https://www.bitchute.com/embed/lbb5G1hjPhw/',
        'only_matching': True,
    }, {
        'url': 'https://www.bitchute.com/torrent/Zee5BE49045h/szoMrox2JEI.webtorrent',
        'only_matching': True,
    }]
    def _real_extract(self, url):
        video_id = self._match_id(url)
        webpage = self._download_webpage(
            'https://www.bitchute.com/video/%s' % video_id, video_id, headers={
                'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.57 Safari/537.36',
            })
        title = self._search_regex(
            (r'<[^>]+\bid=["\']video-title[^>]+>([^<]+)', r'<title>([^<]+)'),
            webpage, 'title', default=None) or self._html_search_meta(
            'description', webpage, 'title',
            default=None) or self._og_search_description(webpage)
        formats = [
            {'url': mobj.group('url')}
            for mobj in re.finditer(
                r'addWebSeed\s*\(\s*(["\'])(?P<url>(?:(?!\1).)+)\1', webpage)]
        self._sort_formats(formats)
        description = self._html_search_regex(
            r'(?s)<div\b[^>]+\bclass=["\']full hidden[^>]+>(.+?)</div>',
            webpage, 'description', fatal=False)
        thumbnail = self._og_search_thumbnail(
            webpage, default=None) or self._html_search_meta(
            'twitter:image:src', webpage, 'thumbnail')
        uploader = self._html_search_regex(
            r'(?s)<p\b[^>]+\bclass=["\']video-author[^>]+>(.+?)</p>', webpage,
            'uploader', fatal=False)
        return {
            'id': video_id,
            'title': title,
            'description': description,
            'thumbnail': thumbnail,
            'uploader': uploader,
            'formats': formats,
        }
 class BitChuteChannelIE(InfoExtractor):
    _VALID_URL = r'https?://(?:www\.)?bitchute\.com/channel/(?P<id>[^/?#&]+)'
    _TEST = {
        'url': 'https://www.bitchute.com/channel/victoriaxrave/',
        'playlist_mincount': 185,
        'info_dict': {
            'id': 'victoriaxrave',
        },
    }
    _TOKEN = 'zyG6tQcGPE5swyAEFLqKUwMuMMuF6IO2DZ6ZDQjGfsL0e4dcTLwqkTTul05Jdve7'
    def _entries(self, channel_id):
        channel_url = 'https://www.bitchute.com/channel/%s/' % channel_id
        offset = 0
        for page_num in itertools.count(1):
            data = self._download_json(
                '%sextend/' % channel_url, channel_id,
                'Downloading channel page %d' % page_num,
                data=urlencode_postdata({
                    'csrfmiddlewaretoken': self._TOKEN,
                    'name': '',
                    'offset': offset,
                }), headers={
                    'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
                    'Referer': channel_url,
                    'X-Requested-With': 'XMLHttpRequest',
                    'Cookie': 'csrftoken=%s' % self._TOKEN,
                })
            if data.get('success') is False:
                break
            html = data.get('html')
            if not html:
                break
            video_ids = re.findall(
                r'class=["\']channel-videos-image-container[^>]+>\s*<a\b[^>]+\bhref=["\']/video/([^"\'/]+)',
                html)
            if not video_ids:
                break
            offset += len(video_ids)
            for video_id in video_ids:
                yield self.url_result(
                    'https://www.bitchute.com/video/%s' % video_id,
                    ie=BitChuteIE.ie_key(), video_id=video_id)
    def _real_extract(self, url):
        channel_id = self._match_id(url)
        return self.playlist_result(
            self._entries(channel_id), playlist_id=channel_id)
--- a/youtube_dl/extractor/breakcom.py
+++ b/youtube_dl/extractor/breakcom.py
@ -3,15 +3,15 @@ from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
-from ..compat import compat_str
+from .youtube import YoutubeIE
 from ..utils import (
    int_or_none,
-    parse_age_limit,
+    url_or_none,
 )
 class BreakIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?(?P<site>break|screenjunkies)\.com/video/(?P<display_id>[^/]+?)(?:-(?P<id>\d+))?(?:[/?#&]|$)'
+    _VALID_URL = r'https?://(?:www\.)?break\.com/video/(?P<display_id>[^/]+?)(?:-(?P<id>\d+))?(?:[/?#&]|$)'
    _TESTS = [{
        'url': 'http://www.break.com/video/when-girls-act-like-guys-2468056',
        'info_dict': {
@ -19,125 +19,73 @@ class BreakIE(InfoExtractor):
            'ext': 'mp4',
            'title': 'When Girls Act Like D-Bags',
            'age_limit': 13,
        },
    }, {
        # youtube embed
        'url': 'http://www.break.com/video/someone-forgot-boat-brakes-work',
        'info_dict': {
            'id': 'RrrDLdeL2HQ',
            'ext': 'mp4',
            'title': 'Whale Watching Boat Crashing Into San Diego Dock',
            'description': 'md5:afc1b2772f0a8468be51dd80eb021069',
            'upload_date': '20160331',
            'uploader': 'Steve Holden',
            'uploader_id': 'sdholden07',
        },
        'params': {
            'skip_download': True,
        }
    }, {
        'url': 'http://www.screenjunkies.com/video/best-quentin-tarantino-movie-2841915',
        'md5': '5c2b686bec3d43de42bde9ec047536b0',
        'info_dict': {
            'id': '2841915',
            'display_id': 'best-quentin-tarantino-movie',
            'ext': 'mp4',
            'title': 'Best Quentin Tarantino Movie',
            'thumbnail': r're:^https?://.*\.jpg',
            'duration': 3671,
            'age_limit': 13,
            'tags': list,
        },
    }, {
        'url': 'http://www.screenjunkies.com/video/honest-trailers-the-dark-knight',
        'info_dict': {
            'id': '2348808',
            'display_id': 'honest-trailers-the-dark-knight',
            'ext': 'mp4',
            'title': 'Honest Trailers - The Dark Knight',
            'thumbnail': r're:^https?://.*\.(?:jpg|png)',
            'age_limit': 10,
            'tags': list,
        },
    }, {
        # requires subscription but worked around
        'url': 'http://www.screenjunkies.com/video/knocking-dead-ep-1-the-show-so-far-3003285',
        'info_dict': {
            'id': '3003285',
            'display_id': 'knocking-dead-ep-1-the-show-so-far',
            'ext': 'mp4',
            'title': 'State of The Dead Recap: Knocking Dead Pilot',
            'thumbnail': r're:^https?://.*\.jpg',
            'duration': 3307,
            'age_limit': 13,
            'tags': list,
        },
    }, {
        'url': 'http://www.break.com/video/ugc/baby-flex-2773063',
        'only_matching': True,
    }]
    _DEFAULT_BITRATES = (48, 150, 320, 496, 864, 2240, 3264)
    def _real_extract(self, url):
-        site, display_id, video_id = re.match(self._VALID_URL, url).groups()
+        display_id, video_id = re.match(self._VALID_URL, url).groups()
-        if not video_id:
+        webpage = self._download_webpage(url, display_id)
            webpage = self._download_webpage(url, display_id)
            video_id = self._search_regex(
                (r'src=["\']/embed/(\d+)', r'data-video-content-id=["\'](\d+)'),
                webpage, 'video id')
-        webpage = self._download_webpage(
+        youtube_url = YoutubeIE._extract_url(webpage)
-            'http://www.%s.com/embed/%s' % (site, video_id),
+        if youtube_url:
-            display_id, 'Downloading video embed page')
+            return self.url_result(youtube_url, ie=YoutubeIE.ie_key())
-        embed_vars = self._parse_json(
+
        content = self._parse_json(
            self._search_regex(
-                r'(?s)embedVars\s*=\s*({.+?})\s*</script>', webpage, 'embed vars'),
+                r'(?s)content["\']\s*:\s*(\[.+?\])\s*[,\n]', webpage,
                'content'),
            display_id)
        youtube_id = embed_vars.get('youtubeId')
        if youtube_id:
            return self.url_result(youtube_id, 'Youtube')
        title = embed_vars['contentName']
        formats = []
-        bitrates = []
+        for video in content:
-        for f in embed_vars.get('media', []):
+            video_url = url_or_none(video.get('url'))
-            if not f.get('uri') or f.get('mediaPurpose') != 'play':
+            if not video_url:
                continue
-            bitrate = int_or_none(f.get('bitRate'))
+            bitrate = int_or_none(self._search_regex(
-            if bitrate:
+                r'(\d+)_kbps', video_url, 'tbr', default=None))
                bitrates.append(bitrate)
            formats.append({
-                'url': f['uri'],
+                'url': video_url,
                'format_id': 'http-%d' % bitrate if bitrate else 'http',
                'width': int_or_none(f.get('width')),
                'height': int_or_none(f.get('height')),
                'tbr': bitrate,
                'format': 'mp4',
            })
        if not bitrates:
            # When subscriptionLevel > 0, i.e. plus subscription is required
            # media list will be empty. However, hds and hls uris are still
            # available. We can grab them assuming bitrates to be default.
            bitrates = self._DEFAULT_BITRATES
        auth_token = embed_vars.get('AuthToken')
        def construct_manifest_url(base_url, ext):
            pieces = [base_url]
            pieces.extend([compat_str(b) for b in bitrates])
            pieces.append('_kbps.mp4.%s?%s' % (ext, auth_token))
            return ','.join(pieces)
        if bitrates and auth_token:
            hds_url = embed_vars.get('hdsUri')
            if hds_url:
                formats.extend(self._extract_f4m_formats(
                    construct_manifest_url(hds_url, 'f4m'),
                    display_id, f4m_id='hds', fatal=False))
            hls_url = embed_vars.get('hlsUri')
            if hls_url:
                formats.extend(self._extract_m3u8_formats(
                    construct_manifest_url(hls_url, 'm3u8'),
                    display_id, 'mp4', entry_protocol='m3u8_native', m3u8_id='hls', fatal=False))
        self._sort_formats(formats)
        title = self._search_regex(
            (r'title["\']\s*:\s*(["\'])(?P<value>(?:(?!\1).)+)\1',
             r'<h1[^>]*>(?P<value>[^<]+)'), webpage, 'title', group='value')
        def get(key, name):
            return int_or_none(self._search_regex(
                r'%s["\']\s*:\s*["\'](\d+)' % key, webpage, name,
                default=None))
        age_limit = get('ratings', 'age limit')
        video_id = video_id or get('pid', 'video id') or display_id
        return {
            'id': video_id,
            'display_id': display_id,
            'title': title,
-            'thumbnail': embed_vars.get('thumbUri'),
+            'thumbnail': self._og_search_thumbnail(webpage),
-            'duration': int_or_none(embed_vars.get('videoLengthInSeconds')) or None,
+            'age_limit': age_limit,
            'age_limit': parse_age_limit(embed_vars.get('audienceRating')),
            'tags': embed_vars.get('tags', '').split(','),
            'formats': formats,
        }
--- a/youtube_dl/extractor/brightcove.py
+++ b/youtube_dl/extractor/brightcove.py
@ -572,7 +572,8 @@ class BrightcoveNewIE(AdobePassIE):
            container = source.get('container')
            ext = mimetype2ext(source.get('type'))
            src = source.get('src')
-            if ext == 'ism' or container == 'WVM':
+            # https://support.brightcove.com/playback-api-video-fields-reference#key_systems_object
            if ext == 'ism' or container == 'WVM' or source.get('key_systems'):
                continue
            elif ext == 'm3u8' or container == 'M2TS':
                if not src:
@ -629,6 +630,14 @@ class BrightcoveNewIE(AdobePassIE):
                        'format_id': build_format_id('rtmp'),
                    })
                formats.append(f)
        if not formats:
            # for sonyliv.com DRM protected videos
            s3_source_url = json_data.get('custom_fields', {}).get('s3sourceurl')
            if s3_source_url:
                formats.append({
                    'url': s3_source_url,
                    'format_id': 'source',
                })
        errors = json_data.get('errors')
        if not formats and errors:
@ -669,7 +678,10 @@ class BrightcoveNewIE(AdobePassIE):
    def _real_extract(self, url):
        url, smuggled_data = unsmuggle_url(url, {})
-        self._initialize_geo_bypass(smuggled_data.get('geo_countries'))
+        self._initialize_geo_bypass({
            'countries': smuggled_data.get('geo_countries'),
            'ip_blocks': smuggled_data.get('geo_ip_blocks'),
        })
        account_id, player_id, embed, video_id = re.match(self._VALID_URL, url).groups()
--- a/youtube_dl/extractor/businessinsider.py
+++ b/youtube_dl/extractor/businessinsider.py
@ -0,0 +1,42 @@
 # coding: utf-8
 from __future__ import unicode_literals
 from .common import InfoExtractor
 from .jwplatform import JWPlatformIE
 class BusinessInsiderIE(InfoExtractor):
    _VALID_URL = r'https?://(?:[^/]+\.)?businessinsider\.(?:com|nl)/(?:[^/]+/)*(?P<id>[^/?#&]+)'
    _TESTS = [{
        'url': 'http://uk.businessinsider.com/how-much-radiation-youre-exposed-to-in-everyday-life-2016-6',
        'md5': 'ca237a53a8eb20b6dc5bd60564d4ab3e',
        'info_dict': {
            'id': 'hZRllCfw',
            'ext': 'mp4',
            'title': "Here's how much radiation you're exposed to in everyday life",
            'description': 'md5:9a0d6e2c279948aadaa5e84d6d9b99bd',
            'upload_date': '20170709',
            'timestamp': 1499606400,
        },
        'params': {
            'skip_download': True,
        },
    }, {
        'url': 'https://www.businessinsider.nl/5-scientifically-proven-things-make-you-less-attractive-2017-7/',
        'only_matching': True,
    }, {
        'url': 'http://www.businessinsider.com/excel-index-match-vlookup-video-how-to-2015-2?IR=T',
        'only_matching': True,
    }]
    def _real_extract(self, url):
        video_id = self._match_id(url)
        webpage = self._download_webpage(url, video_id)
        jwplatform_id = self._search_regex(
            (r'data-media-id=["\']([a-zA-Z0-9]{8})',
             r'id=["\']jwplayer_([a-zA-Z0-9]{8})',
             r'id["\']?\s*:\s*["\']?([a-zA-Z0-9]{8})'),
            webpage, 'jwplatform id')
        return self.url_result(
            'jwplatform:%s' % jwplatform_id, ie=JWPlatformIE.ie_key(),
            video_id=video_id)
--- a/youtube_dl/extractor/cammodels.py
+++ b/youtube_dl/extractor/cammodels.py
@ -0,0 +1,96 @@
 # coding: utf-8
 from __future__ import unicode_literals
 from .common import InfoExtractor
 from ..utils import (
    ExtractorError,
    int_or_none,
    url_or_none,
 )
 class CamModelsIE(InfoExtractor):
    _VALID_URL = r'https?://(?:www\.)?cammodels\.com/cam/(?P<id>[^/?#&]+)'
    _TESTS = [{
        'url': 'https://www.cammodels.com/cam/AutumnKnight/',
        'only_matching': True,
    }]
    def _real_extract(self, url):
        user_id = self._match_id(url)
        webpage = self._download_webpage(
            url, user_id, headers=self.geo_verification_headers())
        manifest_root = self._html_search_regex(
            r'manifestUrlRoot=([^&\']+)', webpage, 'manifest', default=None)
        if not manifest_root:
            ERRORS = (
                ("I'm offline, but let's stay connected", 'This user is currently offline'),
                ('in a private show', 'This user is in a private show'),
                ('is currently performing LIVE', 'This model is currently performing live'),
            )
            for pattern, message in ERRORS:
                if pattern in webpage:
                    error = message
                    expected = True
                    break
            else:
                error = 'Unable to find manifest URL root'
                expected = False
            raise ExtractorError(error, expected=expected)
        manifest = self._download_json(
            '%s%s.json' % (manifest_root, user_id), user_id)
        formats = []
        for format_id, format_dict in manifest['formats'].items():
            if not isinstance(format_dict, dict):
                continue
            encodings = format_dict.get('encodings')
            if not isinstance(encodings, list):
                continue
            vcodec = format_dict.get('videoCodec')
            acodec = format_dict.get('audioCodec')
            for media in encodings:
                if not isinstance(media, dict):
                    continue
                media_url = url_or_none(media.get('location'))
                if not media_url:
                    continue
                format_id_list = [format_id]
                height = int_or_none(media.get('videoHeight'))
                if height is not None:
                    format_id_list.append('%dp' % height)
                f = {
                    'url': media_url,
                    'format_id': '-'.join(format_id_list),
                    'width': int_or_none(media.get('videoWidth')),
                    'height': height,
                    'vbr': int_or_none(media.get('videoKbps')),
                    'abr': int_or_none(media.get('audioKbps')),
                    'fps': int_or_none(media.get('fps')),
                    'vcodec': vcodec,
                    'acodec': acodec,
                }
                if 'rtmp' in format_id:
                    f['ext'] = 'flv'
                elif 'hls' in format_id:
                    f.update({
                        'ext': 'mp4',
                        # hls skips fragments, preferring rtmp
                        'preference': -1,
                    })
                else:
                    continue
                formats.append(f)
        self._sort_formats(formats)
        return {
            'id': user_id,
            'title': self._live_title(user_id),
            'is_live': True,
            'formats': formats,
        }
--- a/youtube_dl/extractor/camtube.py
+++ b/youtube_dl/extractor/camtube.py
@ -0,0 +1,69 @@
 # coding: utf-8
 from __future__ import unicode_literals
 from .common import InfoExtractor
 from ..utils import (
    int_or_none,
    unified_timestamp,
 )
 class CamTubeIE(InfoExtractor):
    _VALID_URL = r'https?://(?:(?:www|api)\.)?camtube\.co/recordings?/(?P<id>[^/?#&]+)'
    _TESTS = [{
        'url': 'https://camtube.co/recording/minafay-030618-1136-chaturbate-female',
        'info_dict': {
            'id': '42ad3956-dd5b-445a-8313-803ea6079fac',
            'display_id': 'minafay-030618-1136-chaturbate-female',
            'ext': 'mp4',
            'title': 'minafay-030618-1136-chaturbate-female',
            'duration': 1274,
            'timestamp': 1528018608,
            'upload_date': '20180603',
        },
        'params': {
            'skip_download': True,
        },
    }]
    _API_BASE = 'https://api.camtube.co'
    def _real_extract(self, url):
        display_id = self._match_id(url)
        token = self._download_json(
            '%s/rpc/session/new' % self._API_BASE, display_id,
            'Downloading session token')['token']
        self._set_cookie('api.camtube.co', 'session', token)
        video = self._download_json(
            '%s/recordings/%s' % (self._API_BASE, display_id), display_id,
            headers={'Referer': url})
        video_id = video['uuid']
        timestamp = unified_timestamp(video.get('createdAt'))
        duration = int_or_none(video.get('duration'))
        view_count = int_or_none(video.get('viewCount'))
        like_count = int_or_none(video.get('likeCount'))
        creator = video.get('stageName')
        formats = [{
            'url': '%s/recordings/%s/manifest.m3u8'
                   % (self._API_BASE, video_id),
            'format_id': 'hls',
            'ext': 'mp4',
            'protocol': 'm3u8_native',
        }]
        return {
            'id': video_id,
            'display_id': display_id,
            'title': display_id,
            'timestamp': timestamp,
            'duration': duration,
            'view_count': view_count,
            'like_count': like_count,
            'creator': creator,
            'formats': formats,
        }
--- a/youtube_dl/extractor/canvas.py
+++ b/youtube_dl/extractor/canvas.py
@ -11,6 +11,7 @@ from ..utils import (
    strip_or_none,
    float_or_none,
    int_or_none,
    merge_dicts,
    parse_iso8601,
 )
@ -248,9 +249,13 @@ class VrtNUIE(GigyaBaseIE):
        webpage, urlh = self._download_webpage_handle(url, display_id)
-        title = self._html_search_regex(
+        info = self._search_json_ld(webpage, display_id, default={})
        # title is optional here since it may be extracted by extractor
        # that is delegated from here
        title = strip_or_none(self._html_search_regex(
            r'(?ms)<h1 class="content__heading">(.+?)</h1>',
-            webpage, 'title').strip()
+            webpage, 'title', default=None))
        description = self._html_search_regex(
            r'(?ms)<div class="content__description">(.+?)</div>',
@ -295,7 +300,7 @@ class VrtNUIE(GigyaBaseIE):
        # the first one
        video_id = list(video.values())[0].get('videoid')
-        return {
+        return merge_dicts(info, {
            '_type': 'url_transparent',
            'url': 'https://mediazone.vrt.be/api/v1/vrtvideo/assets/%s' % video_id,
            'ie_key': CanvasIE.ie_key(),
@ -307,4 +312,4 @@ class VrtNUIE(GigyaBaseIE):
            'season_number': season_number,
            'episode_number': episode_number,
            'release_date': release_date,
-        }
+        })
--- a/youtube_dl/extractor/cbc.py
+++ b/youtube_dl/extractor/cbc.py
@ -5,7 +5,10 @@ import json
 import re
 from .common import InfoExtractor
-from ..compat import compat_str
+from ..compat import (
    compat_str,
    compat_HTTPError,
 )
 from ..utils import (
    js_to_json,
    smuggle_url,
@ -14,9 +17,11 @@ from ..utils import (
    xpath_element,
    xpath_with_ns,
    find_xpath_attr,
    orderedSet,
    parse_duration,
    parse_iso8601,
    parse_age_limit,
    strip_or_none,
    int_or_none,
    ExtractorError,
 )
@ -126,15 +131,23 @@ class CBCIE(InfoExtractor):
    def _real_extract(self, url):
        display_id = self._match_id(url)
        webpage = self._download_webpage(url, display_id)
        title = self._og_search_title(webpage, default=None) or self._html_search_meta(
            'twitter:title', webpage, 'title', default=None) or self._html_search_regex(
                r'<title>([^<]+)</title>', webpage, 'title', fatal=False)
        entries = [
            self._extract_player_init(player_init, display_id)
            for player_init in re.findall(r'CBC\.APP\.Caffeine\.initInstance\(({.+?})\);', webpage)]
        media_ids = []
        for media_id_re in (
                r'<iframe[^>]+src="[^"]+?mediaId=(\d+)"',
                r'<div[^>]+\bid=["\']player-(\d+)',
                r'guid["\']\s*:\s*["\'](\d+)'):
            media_ids.extend(re.findall(media_id_re, webpage))
        entries.extend([
            self.url_result('cbcplayer:%s' % media_id, 'CBCPlayer', media_id)
-            for media_id in re.findall(r'<iframe[^>]+src="[^"]+?mediaId=(\d+)"', webpage)])
+            for media_id in orderedSet(media_ids)])
        return self.playlist_result(
-            entries, display_id,
+            entries, display_id, strip_or_none(title),
            self._og_search_title(webpage, fatal=False),
            self._og_search_description(webpage))
@ -206,30 +219,48 @@ class CBCWatchBaseIE(InfoExtractor):
    def _call_api(self, path, video_id):
        url = path if path.startswith('http') else self._API_BASE_URL + path
-        result = self._download_xml(url, video_id, headers={
+        for _ in range(2):
-            'X-Clearleap-DeviceId': self._device_id,
+            try:
-            'X-Clearleap-DeviceToken': self._device_token,
+                result = self._download_xml(url, video_id, headers={
-        })
+                    'X-Clearleap-DeviceId': self._device_id,
                    'X-Clearleap-DeviceToken': self._device_token,
                })
            except ExtractorError as e:
                if isinstance(e.cause, compat_HTTPError) and e.cause.code == 401:
                    # Device token has expired, re-acquiring device token
                    self._register_device()
                    continue
                raise
        error_message = xpath_text(result, 'userMessage') or xpath_text(result, 'systemMessage')
        if error_message:
            raise ExtractorError('%s said: %s' % (self.IE_NAME, error_message))
        return result
    def _real_initialize(self):
-        if not self._device_id or not self._device_token:
+        if self._valid_device_token():
-            device = self._downloader.cache.load('cbcwatch', 'device') or {}
+            return
-            self._device_id, self._device_token = device.get('id'), device.get('token')
+        device = self._downloader.cache.load('cbcwatch', 'device') or {}
-            if not self._device_id or not self._device_token:
+        self._device_id, self._device_token = device.get('id'), device.get('token')
-                result = self._download_xml(
+        if self._valid_device_token():
-                    self._API_BASE_URL + 'device/register',
+            return
-                    None, data=b'<device><type>web</type></device>')
+        self._register_device()
-                self._device_id = xpath_text(result, 'deviceId', fatal=True)
+
-                self._device_token = xpath_text(result, 'deviceToken', fatal=True)
+    def _valid_device_token(self):
-                self._downloader.cache.store(
+        return self._device_id and self._device_token
-                    'cbcwatch', 'device', {
+
-                        'id': self._device_id,
+    def _register_device(self):
-                        'token': self._device_token,
+        self._device_id = self._device_token = None
-                    })
+        result = self._download_xml(
            self._API_BASE_URL + 'device/register',
            None, 'Acquiring device token',
            data=b'<device><type>web</type></device>')
        self._device_id = xpath_text(result, 'deviceId', fatal=True)
        self._device_token = xpath_text(result, 'deviceToken', fatal=True)
        self._downloader.cache.store(
            'cbcwatch', 'device', {
                'id': self._device_id,
                'token': self._device_token,
            })
    def _parse_rss_feed(self, rss):
        channel = xpath_element(rss, 'channel', fatal=True)
--- a/youtube_dl/extractor/cbs.py
+++ b/youtube_dl/extractor/cbs.py
@ -65,7 +65,7 @@ class CBSIE(CBSBaseIE):
        last_e = None
        for item in items_data.findall('.//item'):
            asset_type = xpath_text(item, 'assetType')
-            if not asset_type or asset_type in asset_types:
+            if not asset_type or asset_type in asset_types or asset_type in ('HLS_FPS', 'DASH_CENC'):
                continue
            asset_types.append(asset_type)
            query = {
--- a/youtube_dl/extractor/cbssports.py
+++ b/youtube_dl/extractor/cbssports.py
@ -4,28 +4,35 @@ from .cbs import CBSBaseIE
 class CBSSportsIE(CBSBaseIE):
-    _VALID_URL = r'https?://(?:www\.)?cbssports\.com/video/player/[^/]+/(?P<id>\d+)'
+    _VALID_URL = r'https?://(?:www\.)?cbssports\.com/[^/]+/(?:video|news)/(?P<id>[^/?#&]+)'
    _TESTS = [{
-        'url': 'http://www.cbssports.com/video/player/videos/708337219968/0/ben-simmons-the-next-lebron?-not-so-fast',
+        'url': 'https://www.cbssports.com/nba/video/donovan-mitchell-flashes-star-potential-in-game-2-victory-over-thunder/',
        'info_dict': {
-            'id': '708337219968',
+            'id': '1214315075735',
            'ext': 'mp4',
-            'title': 'Ben Simmons the next LeBron? Not so fast',
+            'title': 'Donovan Mitchell flashes star potential in Game 2 victory over Thunder',
-            'description': 'md5:854294f627921baba1f4b9a990d87197',
+            'description': 'md5:df6f48622612c2d6bd2e295ddef58def',
-            'timestamp': 1466293740,
+            'timestamp': 1524111457,
-            'upload_date': '20160618',
+            'upload_date': '20180419',
            'uploader': 'CBSI-NEW',
        },
        'params': {
            # m3u8 download
            'skip_download': True,
        }
    }, {
        'url': 'https://www.cbssports.com/nba/news/nba-playoffs-2018-watch-76ers-vs-heat-game-3-series-schedule-tv-channel-online-stream/',
        'only_matching': True,
    }]
    def _extract_video_info(self, filter_query, video_id):
        return self._extract_feed_info('dJ5BDC', 'VxxJg8Ymh8sE', filter_query, video_id)
    def _real_extract(self, url):
-        video_id = self._match_id(url)
+        display_id = self._match_id(url)
        webpage = self._download_webpage(url, display_id)
        video_id = self._search_regex(
            [r'(?:=|%26)pcid%3D(\d+)', r'embedVideo(?:Container)?_(\d+)'],
            webpage, 'video id')
        return self._extract_video_info('byId=%s' % video_id, video_id)
--- a/youtube_dl/extractor/ccma.py
+++ b/youtube_dl/extractor/ccma.py
@ -5,10 +5,12 @@ import re
 from .common import InfoExtractor
 from ..utils import (
    clean_html,
    int_or_none,
    parse_duration,
    parse_iso8601,
-    clean_html,
+    parse_resolution,
    url_or_none,
 )
@ -40,34 +42,42 @@ class CCMAIE(InfoExtractor):
    def _real_extract(self, url):
        media_type, media_id = re.match(self._VALID_URL, url).groups()
-        media_data = {}
+
-        formats = []
+        media = self._download_json(
-        profiles = ['pc'] if media_type == 'audio' else ['mobil', 'pc']
+            'http://dinamics.ccma.cat/pvideo/media.jsp', media_id, query={
        for i, profile in enumerate(profiles):
            md = self._download_json('http://dinamics.ccma.cat/pvideo/media.jsp', media_id, query={
                'media': media_type,
                'idint': media_id,
-                'profile': profile,
+            })
-            }, fatal=False)
+
-            if md:
+        formats = []
-                media_data = md
+        media_url = media['media']['url']
-                media_url = media_data.get('media', {}).get('url')
+        if isinstance(media_url, list):
-                if media_url:
+            for format_ in media_url:
-                    formats.append({
+                format_url = url_or_none(format_.get('file'))
-                        'format_id': profile,
+                if not format_url:
-                        'url': media_url,
+                    continue
-                        'quality': i,
+                label = format_.get('label')
-                    })
+                f = parse_resolution(label)
                f.update({
                    'url': format_url,
                    'format_id': label,
                })
                formats.append(f)
        else:
            formats.append({
                'url': media_url,
                'vcodec': 'none' if media_type == 'audio' else None,
            })
        self._sort_formats(formats)
-        informacio = media_data['informacio']
+        informacio = media['informacio']
        title = informacio['titol']
        durada = informacio.get('durada', {})
        duration = int_or_none(durada.get('milisegons'), 1000) or parse_duration(durada.get('text'))
        timestamp = parse_iso8601(informacio.get('data_emissio', {}).get('utc'))
        subtitles = {}
-        subtitols = media_data.get('subtitols', {})
+        subtitols = media.get('subtitols', {})
        if subtitols:
            sub_url = subtitols.get('url')
            if sub_url:
@ -77,7 +87,7 @@ class CCMAIE(InfoExtractor):
                    })
        thumbnails = []
-        imatges = media_data.get('imatges', {})
+        imatges = media.get('imatges', {})
        if imatges:
            thumbnail_url = imatges.get('url')
            if thumbnail_url:
--- a/youtube_dl/extractor/cda.py
+++ b/youtube_dl/extractor/cda.py
--- a/youtube_dl/extractor/ceskatelevize.py
+++ b/youtube_dl/extractor/ceskatelevize.py
@ -108,7 +108,7 @@ class CeskaTelevizeIE(InfoExtractor):
        for user_agent in (None, USER_AGENTS['Safari']):
            req = sanitized_Request(
-                'http://www.ceskatelevize.cz/ivysilani/ajax/get-client-playlist',
+                'https://www.ceskatelevize.cz/ivysilani/ajax/get-client-playlist',
                data=urlencode_postdata(data))
            req.add_header('Content-type', 'application/x-www-form-urlencoded')
--- a/youtube_dl/extractor/chaturbate.py
+++ b/youtube_dl/extractor/chaturbate.py
@ -31,7 +31,8 @@ class ChaturbateIE(InfoExtractor):
    def _real_extract(self, url):
        video_id = self._match_id(url)
-        webpage = self._download_webpage(url, video_id)
+        webpage = self._download_webpage(
            url, video_id, headers=self.geo_verification_headers())
        m3u8_urls = []
--- a/youtube_dl/extractor/cloudflarestream.py
+++ b/youtube_dl/extractor/cloudflarestream.py
@ -0,0 +1,60 @@
 # coding: utf-8
 from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
 class CloudflareStreamIE(InfoExtractor):
    _VALID_URL = r'''(?x)
                    https?://
                        (?:
                            (?:watch\.)?cloudflarestream\.com/|
                            embed\.cloudflarestream\.com/embed/[^/]+\.js\?.*?\bvideo=
                        )
                        (?P<id>[\da-f]+)
                    '''
    _TESTS = [{
        'url': 'https://embed.cloudflarestream.com/embed/we4g.fla9.latest.js?video=31c9291ab41fac05471db4e73aa11717',
        'info_dict': {
            'id': '31c9291ab41fac05471db4e73aa11717',
            'ext': 'mp4',
            'title': '31c9291ab41fac05471db4e73aa11717',
        },
        'params': {
            'skip_download': True,
        },
    }, {
        'url': 'https://watch.cloudflarestream.com/9df17203414fd1db3e3ed74abbe936c1',
        'only_matching': True,
    }, {
        'url': 'https://cloudflarestream.com/31c9291ab41fac05471db4e73aa11717/manifest/video.mpd',
        'only_matching': True,
    }]
    @staticmethod
    def _extract_urls(webpage):
        return [
            mobj.group('url')
            for mobj in re.finditer(
                r'<script[^>]+\bsrc=(["\'])(?P<url>(?:https?:)?//embed\.cloudflarestream\.com/embed/[^/]+\.js\?.*?\bvideo=[\da-f]+?.*?)\1',
                webpage)]
    def _real_extract(self, url):
        video_id = self._match_id(url)
        formats = self._extract_m3u8_formats(
            'https://cloudflarestream.com/%s/manifest/video.m3u8' % video_id,
            video_id, 'mp4', entry_protocol='m3u8_native', m3u8_id='hls',
            fatal=False)
        formats.extend(self._extract_mpd_formats(
            'https://cloudflarestream.com/%s/manifest/video.mpd' % video_id,
            video_id, mpd_id='dash', fatal=False))
        self._sort_formats(formats)
        return {
            'id': video_id,
            'title': video_id,
            'formats': formats,
        }
--- a/youtube_dl/extractor/clyp.py
+++ b/youtube_dl/extractor/clyp.py
@ -1,15 +1,19 @@
 from __future__ import unicode_literals
 from .common import InfoExtractor
 from ..compat import (
    compat_parse_qs,
    compat_urllib_parse_urlparse,
 )
 from ..utils import (
    float_or_none,
-    parse_iso8601,
+    unified_timestamp,
 )
 class ClypIE(InfoExtractor):
    _VALID_URL = r'https?://(?:www\.)?clyp\.it/(?P<id>[a-z0-9]+)'
-    _TEST = {
+    _TESTS = [{
        'url': 'https://clyp.it/ojz2wfah',
        'md5': '1d4961036c41247ecfdcc439c0cddcbb',
        'info_dict': {
@ -21,13 +25,34 @@ class ClypIE(InfoExtractor):
            'timestamp': 1443515251,
            'upload_date': '20150929',
        },
-    }
+    }, {
        'url': 'https://clyp.it/b04p1odi?token=b0078e077e15835845c528a44417719d',
        'info_dict': {
            'id': 'b04p1odi',
            'ext': 'mp3',
            'title': 'GJ! (Reward Edit)',
            'description': 'Metal Resistance (THE ONE edition)',
            'duration': 177.789,
            'timestamp': 1528241278,
            'upload_date': '20180605',
        },
        'params': {
            'skip_download': True,
        },
    }]
    def _real_extract(self, url):
        audio_id = self._match_id(url)
        qs = compat_parse_qs(compat_urllib_parse_urlparse(url).query)
        token = qs.get('token', [None])[0]
        query = {}
        if token:
            query['token'] = token
        metadata = self._download_json(
-            'https://api.clyp.it/%s' % audio_id, audio_id)
+            'https://api.clyp.it/%s' % audio_id, audio_id, query=query)
        formats = []
        for secure in ('', 'Secure'):
@ -45,7 +70,7 @@ class ClypIE(InfoExtractor):
        title = metadata['Title']
        description = metadata.get('Description')
        duration = float_or_none(metadata.get('Duration'))
-        timestamp = parse_iso8601(metadata.get('DateCreated'))
+        timestamp = unified_timestamp(metadata.get('DateCreated'))
        return {
            'id': audio_id,
--- a/youtube_dl/extractor/common.py
+++ b/youtube_dl/extractor/common.py
@ -19,6 +19,7 @@ from ..compat import (
    compat_cookies,
    compat_etree_fromstring,
    compat_getpass,
    compat_integer_types,
    compat_http_client,
    compat_os_name,
    compat_str,
@ -51,6 +52,7 @@ from ..utils import (
    GeoUtils,
    int_or_none,
    js_to_json,
    JSON_LD_RE,
    mimetype2ext,
    orderedSet,
    parse_codecs,
@ -209,6 +211,11 @@ class InfoExtractor(object):
                    If not explicitly set, calculated from timestamp.
    uploader_id:    Nickname or id of the video uploader.
    uploader_url:   Full URL to a personal webpage of the video uploader.
    channel:        Full name of the channel the video is uploaded on.
                    Note that channel fields may or may not repeat uploader
                    fields. This depends on a particular extractor.
    channel_id:     Id of the channel.
    channel_url:    Full URL to a channel webpage.
    location:       Physical location where the video was filmed.
    subtitles:      The available subtitles as a dictionary in the format
                    {tag: subformats}. "tag" is usually a language code, and
@ -339,15 +346,17 @@ class InfoExtractor(object):
    _GEO_BYPASS attribute may be set to False in order to disable
    geo restriction bypass mechanisms for a particular extractor.
    Though it won't disable explicit geo restriction bypass based on
-    country code provided with geo_bypass_country. (experimental)
+    country code provided with geo_bypass_country.
    _GEO_COUNTRIES attribute may contain a list of presumably geo unrestricted
    countries for this extractor. One of these countries will be used by
    geo restriction bypass mechanism right away in order to bypass
-    geo restriction, of course, if the mechanism is not disabled. (experimental)
+    geo restriction, of course, if the mechanism is not disabled.
-    NB: both these geo attributes are experimental and may change in future
+    _GEO_IP_BLOCKS attribute may contain a list of presumably geo unrestricted
-    or be completely removed.
+    IP blocks in CIDR notation for this extractor. One of these IP blocks
    will be used by geo restriction bypass mechanism similarly
    to _GEO_COUNTRIES.
    Finally, the _WORKING attribute should be set to False for broken IEs
    in order to warn the users and skip the tests.
@ -358,6 +367,7 @@ class InfoExtractor(object):
    _x_forwarded_for_ip = None
    _GEO_BYPASS = True
    _GEO_COUNTRIES = None
    _GEO_IP_BLOCKS = None
    _WORKING = True
    def __init__(self, downloader=None):
@ -392,12 +402,15 @@ class InfoExtractor(object):
    def initialize(self):
        """Initializes an instance (authentication, etc)."""
-        self._initialize_geo_bypass(self._GEO_COUNTRIES)
+        self._initialize_geo_bypass({
            'countries': self._GEO_COUNTRIES,
            'ip_blocks': self._GEO_IP_BLOCKS,
        })
        if not self._ready:
            self._real_initialize()
            self._ready = True
-    def _initialize_geo_bypass(self, countries):
+    def _initialize_geo_bypass(self, geo_bypass_context):
        """
        Initialize geo restriction bypass mechanism.
@ -408,28 +421,82 @@ class InfoExtractor(object):
        HTTP requests.
        This method will be used for initial geo bypass mechanism initialization
-        during the instance initialization with _GEO_COUNTRIES.
+        during the instance initialization with _GEO_COUNTRIES and
        _GEO_IP_BLOCKS.
-        You may also manually call it from extractor's code if geo countries
+        You may also manually call it from extractor's code if geo bypass
        information is not available beforehand (e.g. obtained during
-        extraction) or due to some another reason.
+        extraction) or due to some other reason. In this case you should pass
        this information in geo bypass context passed as first argument. It may
        contain following fields:
        countries:  List of geo unrestricted countries (similar
                    to _GEO_COUNTRIES)
        ip_blocks:  List of geo unrestricted IP blocks in CIDR notation
                    (similar to _GEO_IP_BLOCKS)
        """
        if not self._x_forwarded_for_ip:
-            country_code = self._downloader.params.get('geo_bypass_country', None)
+
-            # If there is no explicit country for geo bypass specified and
+            # Geo bypass mechanism is explicitly disabled by user
-            # the extractor is known to be geo restricted let's fake IP
+            if not self._downloader.params.get('geo_bypass', True):
-            # as X-Forwarded-For right away.
+                return
-            if (not country_code and
+
-                    self._GEO_BYPASS and
+            if not geo_bypass_context:
-                    self._downloader.params.get('geo_bypass', True) and
+                geo_bypass_context = {}
-                    countries):
+
-                country_code = random.choice(countries)
+            # Backward compatibility: previously _initialize_geo_bypass
-            if country_code:
+            # expected a list of countries, some 3rd party code may still use
-                self._x_forwarded_for_ip = GeoUtils.random_ipv4(country_code)
+            # it this way
            if isinstance(geo_bypass_context, (list, tuple)):
                geo_bypass_context = {
                    'countries': geo_bypass_context,
                }
            # The whole point of geo bypass mechanism is to fake IP
            # as X-Forwarded-For HTTP header based on some IP block or
            # country code.
            # Path 1: bypassing based on IP block in CIDR notation
            # Explicit IP block specified by user, use it right away
            # regardless of whether extractor is geo bypassable or not
            ip_block = self._downloader.params.get('geo_bypass_ip_block', None)
            # Otherwise use random IP block from geo bypass context but only
            # if extractor is known as geo bypassable
            if not ip_block:
                ip_blocks = geo_bypass_context.get('ip_blocks')
                if self._GEO_BYPASS and ip_blocks:
                    ip_block = random.choice(ip_blocks)
            if ip_block:
                self._x_forwarded_for_ip = GeoUtils.random_ipv4(ip_block)
                if self._downloader.params.get('verbose', False):
                    self._downloader.to_screen(
                        '[debug] Using fake IP %s as X-Forwarded-For.'
                        % self._x_forwarded_for_ip)
                return
            # Path 2: bypassing based on country code
            # Explicit country code specified by user, use it right away
            # regardless of whether extractor is geo bypassable or not
            country = self._downloader.params.get('geo_bypass_country', None)
            # Otherwise use random country code from geo bypass context but
            # only if extractor is known as geo bypassable
            if not country:
                countries = geo_bypass_context.get('countries')
                if self._GEO_BYPASS and countries:
                    country = random.choice(countries)
            if country:
                self._x_forwarded_for_ip = GeoUtils.random_ipv4(country)
                if self._downloader.params.get('verbose', False):
                    self._downloader.to_screen(
                        '[debug] Using fake IP %s (%s) as X-Forwarded-For.'
-                        % (self._x_forwarded_for_ip, country_code.upper()))
+                        % (self._x_forwarded_for_ip, country.upper()))
    def extract(self, url):
        """Extracts URL information and returns it in list of dicts."""
@ -488,8 +555,26 @@ class InfoExtractor(object):
    def IE_NAME(self):
        return compat_str(type(self).__name__[:-2])
-    def _request_webpage(self, url_or_request, video_id, note=None, errnote=None, fatal=True, data=None, headers={}, query={}):
+    @staticmethod
-        """ Returns the response handle """
+    def __can_accept_status_code(err, expected_status):
        assert isinstance(err, compat_urllib_error.HTTPError)
        if expected_status is None:
            return False
        if isinstance(expected_status, compat_integer_types):
            return err.code == expected_status
        elif isinstance(expected_status, (list, tuple)):
            return err.code in expected_status
        elif callable(expected_status):
            return expected_status(err.code) is True
        else:
            assert False
    def _request_webpage(self, url_or_request, video_id, note=None, errnote=None, fatal=True, data=None, headers={}, query={}, expected_status=None):
        """
        Return the response handle.
        See _download_webpage docstring for arguments specification.
        """
        if note is None:
            self.report_download_webpage(video_id)
        elif note is not False:
@ -518,6 +603,10 @@ class InfoExtractor(object):
        try:
            return self._downloader.urlopen(url_or_request)
        except (compat_urllib_error.URLError, compat_http_client.HTTPException, socket.error) as err:
            if isinstance(err, compat_urllib_error.HTTPError):
                if self.__can_accept_status_code(err, expected_status):
                    return err.fp
            if errnote is False:
                return False
            if errnote is None:
@ -530,13 +619,17 @@ class InfoExtractor(object):
                self._downloader.report_warning(errmsg)
                return False
-    def _download_webpage_handle(self, url_or_request, video_id, note=None, errnote=None, fatal=True, encoding=None, data=None, headers={}, query={}):
+    def _download_webpage_handle(self, url_or_request, video_id, note=None, errnote=None, fatal=True, encoding=None, data=None, headers={}, query={}, expected_status=None):
-        """ Returns a tuple (page content as string, URL handle) """
+        """
        Return a tuple (page content as string, URL handle).
        See _download_webpage docstring for arguments specification.
        """
        # Strip hashes from the URL (#1038)
        if isinstance(url_or_request, (compat_str, str)):
            url_or_request = url_or_request.partition('#')[0]
-        urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query)
+        urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
        if urlh is False:
            assert not fatal
            return False
@ -625,13 +718,52 @@ class InfoExtractor(object):
        return content
-    def _download_webpage(self, url_or_request, video_id, note=None, errnote=None, fatal=True, tries=1, timeout=5, encoding=None, data=None, headers={}, query={}):
+    def _download_webpage(
-        """ Returns the data of the page as a string """
+            self, url_or_request, video_id, note=None, errnote=None,
            fatal=True, tries=1, timeout=5, encoding=None, data=None,
            headers={}, query={}, expected_status=None):
        """
        Return the data of the page as a string.
        Arguments:
        url_or_request -- plain text URL as a string or
            a compat_urllib_request.Requestobject
        video_id -- Video/playlist/item identifier (string)
        Keyword arguments:
        note -- note printed before downloading (string)
        errnote -- note printed in case of an error (string)
        fatal -- flag denoting whether error should be considered fatal,
            i.e. whether it should cause ExtractionError to be raised,
            otherwise a warning will be reported and extraction continued
        tries -- number of tries
        timeout -- sleep interval between tries
        encoding -- encoding for a page content decoding, guessed automatically
            when not explicitly specified
        data -- POST data (bytes)
        headers -- HTTP headers (dict)
        query -- URL query (dict)
        expected_status -- allows to accept failed HTTP requests (non 2xx
            status code) by explicitly specifying a set of accepted status
            codes. Can be any of the following entities:
                - an integer type specifying an exact failed status code to
                  accept
                - a list or a tuple of integer types specifying a list of
                  failed status codes to accept
                - a callable accepting an actual failed status code and
                  returning True if it should be accepted
            Note that this argument does not affect success status codes (2xx)
            which are always accepted.
        """
        success = False
        try_count = 0
        while success is False:
            try:
-                res = self._download_webpage_handle(url_or_request, video_id, note, errnote, fatal, encoding=encoding, data=data, headers=headers, query=query)
+                res = self._download_webpage_handle(
                    url_or_request, video_id, note, errnote, fatal,
                    encoding=encoding, data=data, headers=headers, query=query,
                    expected_status=expected_status)
                success = True
            except compat_http_client.IncompleteRead as e:
                try_count += 1
@ -647,11 +779,17 @@ class InfoExtractor(object):
    def _download_xml_handle(
            self, url_or_request, video_id, note='Downloading XML',
            errnote='Unable to download XML', transform_source=None,
-            fatal=True, encoding=None, data=None, headers={}, query={}):
+            fatal=True, encoding=None, data=None, headers={}, query={},
-        """Return a tuple (xml as an xml.etree.ElementTree.Element, URL handle)"""
+            expected_status=None):
        """
        Return a tuple (xml as an xml.etree.ElementTree.Element, URL handle).
        See _download_webpage docstring for arguments specification.
        """
        res = self._download_webpage_handle(
            url_or_request, video_id, note, errnote, fatal=fatal,
-            encoding=encoding, data=data, headers=headers, query=query)
+            encoding=encoding, data=data, headers=headers, query=query,
            expected_status=expected_status)
        if res is False:
            return res
        xml_string, urlh = res
@ -659,15 +797,21 @@ class InfoExtractor(object):
            xml_string, video_id, transform_source=transform_source,
            fatal=fatal), urlh
-    def _download_xml(self, url_or_request, video_id,
+    def _download_xml(
-                      note='Downloading XML', errnote='Unable to download XML',
+            self, url_or_request, video_id,
-                      transform_source=None, fatal=True, encoding=None,
+            note='Downloading XML', errnote='Unable to download XML',
-                      data=None, headers={}, query={}):
+            transform_source=None, fatal=True, encoding=None,
-        """Return the xml as an xml.etree.ElementTree.Element"""
+            data=None, headers={}, query={}, expected_status=None):
        """
        Return the xml as an xml.etree.ElementTree.Element.
        See _download_webpage docstring for arguments specification.
        """
        res = self._download_xml_handle(
            url_or_request, video_id, note=note, errnote=errnote,
            transform_source=transform_source, fatal=fatal, encoding=encoding,
-            data=data, headers=headers, query=query)
+            data=data, headers=headers, query=query,
            expected_status=expected_status)
        return res if res is False else res[0]
    def _parse_xml(self, xml_string, video_id, transform_source=None, fatal=True):
@ -682,18 +826,43 @@ class InfoExtractor(object):
            else:
                self.report_warning(errmsg + str(ve))
-    def _download_json(self, url_or_request, video_id,
+    def _download_json_handle(
-                       note='Downloading JSON metadata',
+            self, url_or_request, video_id, note='Downloading JSON metadata',
-                       errnote='Unable to download JSON metadata',
+            errnote='Unable to download JSON metadata', transform_source=None,
-                       transform_source=None,
+            fatal=True, encoding=None, data=None, headers={}, query={},
-                       fatal=True, encoding=None, data=None, headers={}, query={}):
+            expected_status=None):
-        json_string = self._download_webpage(
+        """
        Return a tuple (JSON object, URL handle).
        See _download_webpage docstring for arguments specification.
        """
        res = self._download_webpage_handle(
            url_or_request, video_id, note, errnote, fatal=fatal,
-            encoding=encoding, data=data, headers=headers, query=query)
+            encoding=encoding, data=data, headers=headers, query=query,
-        if (not fatal) and json_string is False:
+            expected_status=expected_status)
-            return None
+        if res is False:
            return res
        json_string, urlh = res
        return self._parse_json(
-            json_string, video_id, transform_source=transform_source, fatal=fatal)
+            json_string, video_id, transform_source=transform_source,
            fatal=fatal), urlh
    def _download_json(
            self, url_or_request, video_id, note='Downloading JSON metadata',
            errnote='Unable to download JSON metadata', transform_source=None,
            fatal=True, encoding=None, data=None, headers={}, query={},
            expected_status=None):
        """
        Return the JSON object as a dict.
        See _download_webpage docstring for arguments specification.
        """
        res = self._download_json_handle(
            url_or_request, video_id, note=note, errnote=errnote,
            transform_source=transform_source, fatal=fatal, encoding=encoding,
            data=data, headers=headers, query=query,
            expected_status=expected_status)
        return res if res is False else res[0]
    def _parse_json(self, json_string, video_id, transform_source=None, fatal=True):
        if transform_source:
@ -986,8 +1155,7 @@ class InfoExtractor(object):
    def _search_json_ld(self, html, video_id, expected_type=None, **kwargs):
        json_ld = self._search_regex(
-            r'(?s)<script[^>]+type=(["\'])application/ld\+json\1[^>]*>(?P<json_ld>.+?)</script>',
+            JSON_LD_RE, html, 'JSON-LD', group='json_ld', **kwargs)
            html, 'JSON-LD', group='json_ld', **kwargs)
        default = kwargs.get('default', NO_DEFAULT)
        if not json_ld:
            return default if default is not NO_DEFAULT else {}
@ -1008,6 +1176,40 @@ class InfoExtractor(object):
        if isinstance(json_ld, dict):
            json_ld = [json_ld]
        INTERACTION_TYPE_MAP = {
            'CommentAction': 'comment',
            'AgreeAction': 'like',
            'DisagreeAction': 'dislike',
            'LikeAction': 'like',
            'DislikeAction': 'dislike',
            'ListenAction': 'view',
            'WatchAction': 'view',
            'ViewAction': 'view',
        }
        def extract_interaction_statistic(e):
            interaction_statistic = e.get('interactionStatistic')
            if not isinstance(interaction_statistic, list):
                return
            for is_e in interaction_statistic:
                if not isinstance(is_e, dict):
                    continue
                if is_e.get('@type') != 'InteractionCounter':
                    continue
                interaction_type = is_e.get('interactionType')
                if not isinstance(interaction_type, compat_str):
                    continue
                interaction_count = int_or_none(is_e.get('userInteractionCount'))
                if interaction_count is None:
                    continue
                count_kind = INTERACTION_TYPE_MAP.get(interaction_type.split('/')[-1])
                if not count_kind:
                    continue
                count_key = '%s_count' % count_kind
                if info.get(count_key) is not None:
                    continue
                info[count_key] = interaction_count
        def extract_video_object(e):
            assert e['@type'] == 'VideoObject'
            info.update({
@ -1023,9 +1225,10 @@ class InfoExtractor(object):
                'height': int_or_none(e.get('height')),
                'view_count': int_or_none(e.get('interactionCount')),
            })
            extract_interaction_statistic(e)
        for e in json_ld:
-            if e.get('@context') == 'http://schema.org':
+            if isinstance(e.get('@context'), compat_str) and re.match(r'^https?://schema.org/?$', e.get('@context')):
                item_type = e.get('@type')
                if expected_type is not None and expected_type != item_type:
                    return info
@ -1503,9 +1706,9 @@ class InfoExtractor(object):
                # However, this is not always respected, for example, [2]
                # contains EXT-X-STREAM-INF tag which references AUDIO
                # rendition group but does not have CODECS and despite
-                # referencing audio group an audio group, it represents
+                # referencing an audio group it represents a complete
-                # a complete (with audio and video) format. So, for such cases
+                # (with audio and video) format. So, for such cases we will
-                # we will ignore references to rendition groups and treat them
+                # ignore references to rendition groups and treat them
                # as complete formats.
                if audio_group_id and codecs and f.get('vcodec') != 'none':
                    audio_group = groups.get(audio_group_id)
@ -1661,9 +1864,7 @@ class InfoExtractor(object):
                        'height': height,
                    })
                formats.extend(m3u8_formats)
-                continue
+            elif src_ext == 'f4m':
            if src_ext == 'f4m':
                f4m_url = src_url
                if not f4m_params:
                    f4m_params = {
@ -1673,9 +1874,13 @@ class InfoExtractor(object):
                f4m_url += '&' if '?' in f4m_url else '?'
                f4m_url += compat_urllib_parse_urlencode(f4m_params)
                formats.extend(self._extract_f4m_formats(f4m_url, video_id, f4m_id='hds', fatal=False))
-                continue
+            elif src_ext == 'mpd':
-
+                formats.extend(self._extract_mpd_formats(
-            if src_url.startswith('http') and self._is_valid_url(src, video_id):
+                    src_url, video_id, mpd_id='dash', fatal=False))
            elif re.search(r'\.ism/[Mm]anifest', src_url):
                formats.extend(self._extract_ism_formats(
                    src_url, video_id, ism_id='mss', fatal=False))
            elif src_url.startswith('http') and self._is_valid_url(src, video_id):
                http_count += 1
                formats.append({
                    'url': src_url,
@ -1686,7 +1891,6 @@ class InfoExtractor(object):
                    'width': width,
                    'height': height,
                })
                continue
        return formats
@ -1908,7 +2112,21 @@ class InfoExtractor(object):
                        representation_ms_info = extract_multisegment_info(representation, adaption_set_ms_info)
                        def prepare_template(template_name, identifiers):
-                            t = representation_ms_info[template_name]
+                            tmpl = representation_ms_info[template_name]
                            # First of, % characters outside $...$ templates
                            # must be escaped by doubling for proper processing
                            # by % operator string formatting used further (see
                            # https://github.com/rg3/youtube-dl/issues/16867).
                            t = ''
                            in_template = False
                            for c in tmpl:
                                t += c
                                if c == '$':
                                    in_template = not in_template
                                elif c == '%' and not in_template:
                                    t += c
                            # Next, $...$ templates are translated to their
                            # %(...) counterparts to be used with % operator
                            t = t.replace('$RepresentationID$', representation_id)
                            t = re.sub(r'\$(%s)\$' % '|'.join(identifiers), r'%(\1)d', t)
                            t = re.sub(r'\$(%s)%%([^$]+)\$' % '|'.join(identifiers), r'%(\1)\2', t)
@ -2239,6 +2457,8 @@ class InfoExtractor(object):
                        media_info['subtitles'].setdefault(lang, []).append({
                            'url': absolute_url(src),
                        })
            for f in media_info['formats']:
                f.setdefault('http_headers', {})['Referer'] = base_url
            if media_info['formats'] or media_info['subtitles']:
                entries.append(media_info)
        return entries
--- a/youtube_dl/extractor/crackle.py
+++ b/youtube_dl/extractor/crackle.py
@ -4,23 +4,21 @@ from __future__ import unicode_literals, division
 import re
 from .common import InfoExtractor
-from ..compat import (
+from ..compat import compat_HTTPError
    compat_str,
    compat_HTTPError,
 )
 from ..utils import (
    determine_ext,
    float_or_none,
    int_or_none,
    parse_age_limit,
    parse_duration,
    url_or_none,
    ExtractorError
 )
 class CrackleIE(InfoExtractor):
-    _VALID_URL = r'(?:crackle:|https?://(?:(?:www|m)\.)?crackle\.com/(?:playlist/\d+/|(?:[^/]+/)+))(?P<id>\d+)'
+    _VALID_URL = r'(?:crackle:|https?://(?:(?:www|m)\.)?(?:sony)?crackle\.com/(?:playlist/\d+/|(?:[^/]+/)+))(?P<id>\d+)'
-    _TEST = {
+    _TESTS = [{
        # geo restricted to CA
        'url': 'https://www.crackle.com/andromeda/2502343',
        'info_dict': {
@ -45,7 +43,10 @@ class CrackleIE(InfoExtractor):
            # m3u8 download
            'skip_download': True,
        }
-    }
+    }, {
        'url': 'https://www.sonycrackle.com/andromeda/2502343',
        'only_matching': True,
    }]
    def _real_extract(self, url):
        video_id = self._match_id(url)
@ -83,8 +84,8 @@ class CrackleIE(InfoExtractor):
            for e in media['MediaURLs']:
                if e.get('UseDRM') is True:
                    continue
-                format_url = e.get('Path')
+                format_url = url_or_none(e.get('Path'))
-                if not format_url or not isinstance(format_url, compat_str):
+                if not format_url:
                    continue
                ext = determine_ext(format_url)
                if ext == 'm3u8':
@ -121,8 +122,8 @@ class CrackleIE(InfoExtractor):
                for cc_file in cc_files:
                    if not isinstance(cc_file, dict):
                        continue
-                    cc_url = cc_file.get('Path')
+                    cc_url = url_or_none(cc_file.get('Path'))
-                    if not cc_url or not isinstance(cc_url, compat_str):
+                    if not cc_url:
                        continue
                    lang = cc_file.get('Locale') or 'en'
                    subtitles.setdefault(lang, []).append({'url': cc_url})
--- a/youtube_dl/extractor/crunchyroll.py
+++ b/youtube_dl/extractor/crunchyroll.py
@ -8,6 +8,7 @@ import zlib
 from hashlib import sha1
 from math import pow, sqrt, floor
 from .common import InfoExtractor
 from .vrv import VRVIE
 from ..compat import (
    compat_b64decode,
    compat_etree_fromstring,
@ -18,6 +19,8 @@ from ..compat import (
 from ..utils import (
    ExtractorError,
    bytes_to_intlist,
    extract_attributes,
    float_or_none,
    intlist_to_bytes,
    int_or_none,
    lowercase_escape,
@ -26,7 +29,6 @@ from ..utils import (
    unified_strdate,
    urlencode_postdata,
    xpath_text,
    extract_attributes,
 )
 from ..aes import (
    aes_cbc_decrypt,
@ -43,13 +45,13 @@ class CrunchyrollBaseIE(InfoExtractor):
        data['req'] = 'RpcApi' + method
        data = compat_urllib_parse_urlencode(data).encode('utf-8')
        return self._download_xml(
-            'http://www.crunchyroll.com/xml/',
+            'https://www.crunchyroll.com/xml/',
            video_id, note, fatal=False, data=data, headers={
                'Content-Type': 'application/x-www-form-urlencoded',
            })
    def _login(self):
-        (username, password) = self._get_login_info()
+        username, password = self._get_login_info()
        if username is None:
            return
@ -139,7 +141,8 @@ class CrunchyrollBaseIE(InfoExtractor):
            parsed_url._replace(query=compat_urllib_parse_urlencode(qs, True)))
-class CrunchyrollIE(CrunchyrollBaseIE):
+class CrunchyrollIE(CrunchyrollBaseIE, VRVIE):
    IE_NAME = 'crunchyroll'
    _VALID_URL = r'https?://(?:(?P<prefix>www|m)\.)?(?P<url>crunchyroll\.(?:com|fr)/(?:media(?:-|/\?id=)|[^/]*/[^/?&]*?)(?P<video_id>[0-9]+))(?:[/?&]|$)'
    _TESTS = [{
        'url': 'http://www.crunchyroll.com/wanna-be-the-strongest-in-the-world/episode-1-an-idol-wrestler-is-born-645513',
@ -148,7 +151,7 @@ class CrunchyrollIE(CrunchyrollBaseIE):
            'ext': 'mp4',
            'title': 'Wanna be the Strongest in the World Episode 1 – An Idol-Wrestler is Born!',
            'description': 'md5:2d17137920c64f2f49981a7797d275ef',
-            'thumbnail': 'http://img1.ak.crunchyroll.com/i/spire1-tmb/20c6b5e10f1a47b10516877d3c039cae1380951166_full.jpg',
+            'thumbnail': r're:^https?://.*\.jpg$',
            'uploader': 'Yomiuri Telecasting Corporation (YTV)',
            'upload_date': '20131013',
            'url': 're:(?!.*&amp)',
@ -221,7 +224,7 @@ class CrunchyrollIE(CrunchyrollBaseIE):
        'info_dict': {
            'id': '535080',
            'ext': 'mp4',
-            'title': '11eyes Episode 1 – Piros éjszaka - Red Night',
+            'title': '11eyes Episode 1 – Red Night ~ Piros éjszaka',
            'description': 'Kakeru and Yuka are thrown into an alternate nightmarish world they call "Red Night".',
            'uploader': 'Marvelous AQL Inc.',
            'upload_date': '20091021',
@ -262,6 +265,9 @@ class CrunchyrollIE(CrunchyrollBaseIE):
            # Just test metadata extraction
            'skip_download': True,
        },
    }, {
        'url': 'http://www.crunchyroll.com/media-723735',
        'only_matching': True,
    }]
    _FORMAT_IDS = {
@ -434,13 +440,22 @@ Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
        if 'To view this, please log in to verify you are 18 or older.' in webpage:
            self.raise_login_required()
        media = self._parse_json(self._search_regex(
            r'vilos\.config\.media\s*=\s*({.+?});',
            webpage, 'vilos media', default='{}'), video_id)
        media_metadata = media.get('metadata') or {}
        language = self._search_regex(
            r'(?:vilos\.config\.player\.language|LOCALE)\s*=\s*(["\'])(?P<lang>(?:(?!\1).)+)\1',
            webpage, 'language', default=None, group='lang')
        video_title = self._html_search_regex(
            r'(?s)<h1[^>]*>((?:(?!<h1).)*?<span[^>]+itemprop=["\']title["\'][^>]*>(?:(?!<h1).)+?)</h1>',
            webpage, 'video_title')
        video_title = re.sub(r' {2,}', ' ', video_title)
-        video_description = self._parse_json(self._html_search_regex(
+        video_description = (self._parse_json(self._html_search_regex(
            r'<script[^>]*>\s*.+?\[media_id=%s\].+?({.+?"description"\s*:.+?})\);' % video_id,
-            webpage, 'description', default='{}'), video_id).get('description')
+            webpage, 'description', default='{}'), video_id) or media_metadata).get('description')
        if video_description:
            video_description = lowercase_escape(video_description.replace(r'\r\n', '\n'))
        video_upload_date = self._html_search_regex(
@ -453,92 +468,113 @@ Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
            [r'<a[^>]+href="/publisher/[^"]+"[^>]*>([^<]+)</a>', r'<div>\s*Publisher:\s*<span>\s*(.+?)\s*</span>\s*</div>'],
            webpage, 'video_uploader', fatal=False)
        available_fmts = []
        for a, fmt in re.findall(r'(<a[^>]+token=["\']showmedia\.([0-9]{3,4})p["\'][^>]+>)', webpage):
            attrs = extract_attributes(a)
            href = attrs.get('href')
            if href and '/freetrial' in href:
                continue
            available_fmts.append(fmt)
        if not available_fmts:
            for p in (r'token=["\']showmedia\.([0-9]{3,4})p"', r'showmedia\.([0-9]{3,4})p'):
                available_fmts = re.findall(p, webpage)
                if available_fmts:
                    break
        video_encode_ids = []
        formats = []
-        for fmt in available_fmts:
+        for stream in media.get('streams', []):
-            stream_quality, stream_format = self._FORMAT_IDS[fmt]
+            audio_lang = stream.get('audio_lang')
-            video_format = fmt + 'p'
+            hardsub_lang = stream.get('hardsub_lang')
-            stream_infos = []
+            vrv_formats = self._extract_vrv_formats(
-            streamdata = self._call_rpc_api(
+                stream.get('url'), video_id, stream.get('format'),
-                'VideoPlayer_GetStandardConfig', video_id,
+                audio_lang, hardsub_lang)
-                'Downloading media info for %s' % video_format, data={
+            for f in vrv_formats:
-                    'media_id': video_id,
+                if not hardsub_lang:
-                    'video_format': stream_format,
+                    f['preference'] = 1
-                    'video_quality': stream_quality,
+                language_preference = 0
-                    'current_page': url,
+                if audio_lang == language:
-                })
+                    language_preference += 1
-            if streamdata is not None:
+                if hardsub_lang == language:
-                stream_info = streamdata.find('./{default}preload/stream_info')
+                    language_preference += 1
                if language_preference:
                    f['language_preference'] = language_preference
            formats.extend(vrv_formats)
        if not formats:
            available_fmts = []
            for a, fmt in re.findall(r'(<a[^>]+token=["\']showmedia\.([0-9]{3,4})p["\'][^>]+>)', webpage):
                attrs = extract_attributes(a)
                href = attrs.get('href')
                if href and '/freetrial' in href:
                    continue
                available_fmts.append(fmt)
            if not available_fmts:
                for p in (r'token=["\']showmedia\.([0-9]{3,4})p"', r'showmedia\.([0-9]{3,4})p'):
                    available_fmts = re.findall(p, webpage)
                    if available_fmts:
                        break
            if not available_fmts:
                available_fmts = self._FORMAT_IDS.keys()
            video_encode_ids = []
            for fmt in available_fmts:
                stream_quality, stream_format = self._FORMAT_IDS[fmt]
                video_format = fmt + 'p'
                stream_infos = []
                streamdata = self._call_rpc_api(
                    'VideoPlayer_GetStandardConfig', video_id,
                    'Downloading media info for %s' % video_format, data={
                        'media_id': video_id,
                        'video_format': stream_format,
                        'video_quality': stream_quality,
                        'current_page': url,
                    })
                if streamdata is not None:
                    stream_info = streamdata.find('./{default}preload/stream_info')
                    if stream_info is not None:
                        stream_infos.append(stream_info)
                stream_info = self._call_rpc_api(
                    'VideoEncode_GetStreamInfo', video_id,
                    'Downloading stream info for %s' % video_format, data={
                        'media_id': video_id,
                        'video_format': stream_format,
                        'video_encode_quality': stream_quality,
                    })
                if stream_info is not None:
                    stream_infos.append(stream_info)
-            stream_info = self._call_rpc_api(
+                for stream_info in stream_infos:
-                'VideoEncode_GetStreamInfo', video_id,
+                    video_encode_id = xpath_text(stream_info, './video_encode_id')
-                'Downloading stream info for %s' % video_format, data={
+                    if video_encode_id in video_encode_ids:
-                    'media_id': video_id,
+                        continue
-                    'video_format': stream_format,
+                    video_encode_ids.append(video_encode_id)
                    'video_encode_quality': stream_quality,
                })
            if stream_info is not None:
                stream_infos.append(stream_info)
            for stream_info in stream_infos:
                video_encode_id = xpath_text(stream_info, './video_encode_id')
                if video_encode_id in video_encode_ids:
                    continue
                video_encode_ids.append(video_encode_id)
-                video_file = xpath_text(stream_info, './file')
+                    video_file = xpath_text(stream_info, './file')
-                if not video_file:
+                    if not video_file:
-                    continue
+                        continue
-                if video_file.startswith('http'):
+                    if video_file.startswith('http'):
-                    formats.extend(self._extract_m3u8_formats(
+                        formats.extend(self._extract_m3u8_formats(
-                        video_file, video_id, 'mp4', entry_protocol='m3u8_native',
+                            video_file, video_id, 'mp4', entry_protocol='m3u8_native',
-                        m3u8_id='hls', fatal=False))
+                            m3u8_id='hls', fatal=False))
                    continue
                video_url = xpath_text(stream_info, './host')
                if not video_url:
                    continue
                metadata = stream_info.find('./metadata')
                format_info = {
                    'format': video_format,
                    'height': int_or_none(xpath_text(metadata, './height')),
                    'width': int_or_none(xpath_text(metadata, './width')),
                }
                if '.fplive.net/' in video_url:
                    video_url = re.sub(r'^rtmpe?://', 'http://', video_url.strip())
                    parsed_video_url = compat_urlparse.urlparse(video_url)
                    direct_video_url = compat_urlparse.urlunparse(parsed_video_url._replace(
                        netloc='v.lvlt.crcdn.net',
                        path='%s/%s' % (remove_end(parsed_video_url.path, '/'), video_file.split(':')[-1])))
                    if self._is_valid_url(direct_video_url, video_id, video_format):
                        format_info.update({
                            'format_id': 'http-' + video_format,
                            'url': direct_video_url,
                        })
                        formats.append(format_info)
                        continue
-                format_info.update({
+                    video_url = xpath_text(stream_info, './host')
-                    'format_id': 'rtmp-' + video_format,
+                    if not video_url:
-                    'url': video_url,
+                        continue
-                    'play_path': video_file,
+                    metadata = stream_info.find('./metadata')
-                    'ext': 'flv',
+                    format_info = {
-                })
+                        'format': video_format,
-                formats.append(format_info)
+                        'height': int_or_none(xpath_text(metadata, './height')),
-        self._sort_formats(formats, ('height', 'width', 'tbr', 'fps'))
+                        'width': int_or_none(xpath_text(metadata, './width')),
                    }
                    if '.fplive.net/' in video_url:
                        video_url = re.sub(r'^rtmpe?://', 'http://', video_url.strip())
                        parsed_video_url = compat_urlparse.urlparse(video_url)
                        direct_video_url = compat_urlparse.urlunparse(parsed_video_url._replace(
                            netloc='v.lvlt.crcdn.net',
                            path='%s/%s' % (remove_end(parsed_video_url.path, '/'), video_file.split(':')[-1])))
                        if self._is_valid_url(direct_video_url, video_id, video_format):
                            format_info.update({
                                'format_id': 'http-' + video_format,
                                'url': direct_video_url,
                            })
                            formats.append(format_info)
                            continue
                    format_info.update({
                        'format_id': 'rtmp-' + video_format,
                        'url': video_url,
                        'play_path': video_file,
                        'ext': 'flv',
                    })
                    formats.append(format_info)
        self._sort_formats(formats, ('preference', 'language_preference', 'height', 'width', 'tbr', 'fps'))
        metadata = self._call_rpc_api(
            'VideoPlayer_GetMediaMetadata', video_id,
@ -546,7 +582,17 @@ Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
                'media_id': video_id,
            })
-        subtitles = self.extract_subtitles(video_id, webpage)
+        subtitles = {}
        for subtitle in media.get('subtitles', []):
            subtitle_url = subtitle.get('url')
            if not subtitle_url:
                continue
            subtitles.setdefault(subtitle.get('language', 'enUS'), []).append({
                'url': subtitle_url,
                'ext': subtitle.get('format', 'ass'),
            })
        if not subtitles:
            subtitles = self.extract_subtitles(video_id, webpage)
        # webpage provide more accurate data than series_title from XML
        series = self._html_search_regex(
@ -554,8 +600,8 @@ Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
            webpage, 'series', fatal=False)
        season = xpath_text(metadata, 'series_title')
-        episode = xpath_text(metadata, 'episode_title')
+        episode = xpath_text(metadata, 'episode_title') or media_metadata.get('title')
-        episode_number = int_or_none(xpath_text(metadata, 'episode_number'))
+        episode_number = int_or_none(xpath_text(metadata, 'episode_number') or media_metadata.get('episode_number'))
        season_number = int_or_none(self._search_regex(
            r'(?s)<h\d[^>]+id=["\']showmedia_about_episode_num[^>]+>.+?</h\d>\s*<h4>\s*Season (\d+)',
@ -565,7 +611,8 @@ Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
            'id': video_id,
            'title': video_title,
            'description': video_description,
-            'thumbnail': xpath_text(metadata, 'episode_image_url'),
+            'duration': float_or_none(media_metadata.get('duration'), 1000),
            'thumbnail': xpath_text(metadata, 'episode_image_url') or media_metadata.get('thumbnail', {}).get('url'),
            'uploader': video_uploader,
            'upload_date': video_upload_date,
            'series': series,
@ -580,7 +627,7 @@ Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
 class CrunchyrollShowPlaylistIE(CrunchyrollBaseIE):
    IE_NAME = 'crunchyroll:playlist'
-    _VALID_URL = r'https?://(?:(?P<prefix>www|m)\.)?(?P<url>crunchyroll\.com/(?!(?:news|anime-news|library|forum|launchcalendar|lineup|store|comics|freetrial|login))(?P<id>[\w\-]+))/?(?:\?|$)'
+    _VALID_URL = r'https?://(?:(?P<prefix>www|m)\.)?(?P<url>crunchyroll\.com/(?!(?:news|anime-news|library|forum|launchcalendar|lineup|store|comics|freetrial|login|media-\d+))(?P<id>[\w\-]+))/?(?:\?|$)'
    _TESTS = [{
        'url': 'http://www.crunchyroll.com/a-bridge-to-the-starry-skies-hoshizora-e-kakaru-hashi',
--- a/youtube_dl/extractor/ctvnews.py
+++ b/youtube_dl/extractor/ctvnews.py
@ -11,10 +11,10 @@ class CTVNewsIE(InfoExtractor):
    _VALID_URL = r'https?://(?:.+?\.)?ctvnews\.ca/(?:video\?(?:clip|playlist|bin)Id=|.*?)(?P<id>[0-9.]+)'
    _TESTS = [{
        'url': 'http://www.ctvnews.ca/video?clipId=901995',
-        'md5': '10deb320dc0ccb8d01d34d12fc2ea672',
+        'md5': '9b8624ba66351a23e0b6e1391971f9af',
        'info_dict': {
            'id': '901995',
-            'ext': 'mp4',
+            'ext': 'flv',
            'title': 'Extended: \'That person cannot be me\' Johnson says',
            'description': 'md5:958dd3b4f5bbbf0ed4d045c790d89285',
            'timestamp': 1467286284,
--- a/youtube_dl/extractor/curiositystream.py
+++ b/youtube_dl/extractor/curiositystream.py
@ -35,7 +35,7 @@ class CuriosityStreamBaseIE(InfoExtractor):
        return result['data']
    def _real_initialize(self):
-        (email, password) = self._get_login_info()
+        email, password = self._get_login_info()
        if email is None:
            return
        result = self._download_json(
--- a/youtube_dl/extractor/cwtv.py
+++ b/youtube_dl/extractor/cwtv.py
@ -4,7 +4,10 @@ from __future__ import unicode_literals
 from .common import InfoExtractor
 from ..utils import (
    int_or_none,
    parse_age_limit,
    parse_iso8601,
    smuggle_url,
    str_or_none,
 )
@ -40,10 +43,15 @@ class CWTVIE(InfoExtractor):
            'duration': 1263,
            'series': 'Whose Line Is It Anyway?',
            'season_number': 11,
            'season': '11',
            'episode_number': 20,
            'upload_date': '20151006',
            'timestamp': 1444107300,
            'age_limit': 14,
            'uploader': 'CWTV',
        },
        'params': {
            # m3u8 download
            'skip_download': True,
        },
    }, {
        'url': 'http://cwtv.com/thecw/chroniclesofcisco/?play=8adebe35-f447-465f-ab52-e863506ff6d6',
@ -58,60 +66,28 @@ class CWTVIE(InfoExtractor):
    def _real_extract(self, url):
        video_id = self._match_id(url)
-        video_data = None
+        video_data = self._download_json(
-        formats = []
+            'http://images.cwtv.com/feed/mobileapp/video-meta/apiversion_8/guid_' + video_id,
-        for partner in (154, 213):
+            video_id)['video']
-            vdata = self._download_json(
+        title = video_data['title']
-                'http://metaframe.digitalsmiths.tv/v2/CWtv/assets/%s/partner/%d?format=json' % (video_id, partner), video_id, fatal=False)
+        mpx_url = video_data.get('mpx_url') or 'http://link.theplatform.com/s/cwtv/media/guid/2703454149/%s?formats=M3U' % video_id
            if not vdata:
                continue
            video_data = vdata
            for quality, quality_data in vdata.get('videos', {}).items():
                quality_url = quality_data.get('uri')
                if not quality_url:
                    continue
                if quality == 'variantplaylist':
                    formats.extend(self._extract_m3u8_formats(
                        quality_url, video_id, 'mp4', m3u8_id='hls', fatal=False))
                else:
                    tbr = int_or_none(quality_data.get('bitrate'))
                    format_id = 'http' + ('-%d' % tbr if tbr else '')
                    if self._is_valid_url(quality_url, video_id, format_id):
                        formats.append({
                            'format_id': format_id,
                            'url': quality_url,
                            'tbr': tbr,
                        })
        video_metadata = video_data['assetFields']
        ism_url = video_metadata.get('smoothStreamingUrl')
        if ism_url:
            formats.extend(self._extract_ism_formats(
                ism_url, video_id, ism_id='mss', fatal=False))
        self._sort_formats(formats)
-        thumbnails = [{
+        season = str_or_none(video_data.get('season'))
-            'url': image['uri'],
+        episode = str_or_none(video_data.get('episode'))
-            'width': image.get('width'),
+        if episode and season:
-            'height': image.get('height'),
+            episode = episode.lstrip(season)
        } for image_id, image in video_data['images'].items() if image.get('uri')] if video_data.get('images') else None
        subtitles = {
            'en': [{
                'url': video_metadata['UnicornCcUrl'],
            }],
        } if video_metadata.get('UnicornCcUrl') else None
        return {
            '_type': 'url_transparent',
            'id': video_id,
-            'title': video_metadata['title'],
+            'title': title,
-            'description': video_metadata.get('description'),
+            'url': smuggle_url(mpx_url, {'force_smil_url': True}),
-            'duration': int_or_none(video_metadata.get('duration')),
+            'description': video_data.get('description_long'),
-            'series': video_metadata.get('seriesName'),
+            'duration': int_or_none(video_data.get('duration_secs')),
-            'season_number': int_or_none(video_metadata.get('seasonNumber')),
+            'series': video_data.get('series_name'),
-            'season': video_metadata.get('seasonName'),
+            'season_number': int_or_none(season),
-            'episode_number': int_or_none(video_metadata.get('episodeNumber')),
+            'episode_number': int_or_none(episode),
-            'timestamp': parse_iso8601(video_data.get('startTime')),
+            'timestamp': parse_iso8601(video_data.get('start_time')),
-            'thumbnails': thumbnails,
+            'age_limit': parse_age_limit(video_data.get('rating')),
-            'formats': formats,
+            'ie_key': 'ThePlatform',
            'subtitles': subtitles,
        }
--- a/youtube_dl/extractor/dailymotion.py
+++ b/youtube_dl/extractor/dailymotion.py
@ -1,22 +1,32 @@
 # coding: utf-8
 from __future__ import unicode_literals
-import re
+import base64
-import json
+import functools
 import hashlib
 import itertools
 import json
 import random
 import re
 import string
 from .common import InfoExtractor
-
+from ..compat import compat_struct_pack
 from ..utils import (
    determine_ext,
    error_to_compat_str,
    ExtractorError,
    int_or_none,
    mimetype2ext,
    OnDemandPagedList,
    parse_iso8601,
    sanitized_Request,
    str_to_int,
    try_get,
    unescapeHTML,
-    mimetype2ext,
+    update_url_query,
    url_or_none,
    urlencode_postdata,
 )
@ -64,7 +74,6 @@ class DailymotionIE(DailymotionBaseInfoExtractor):
            'uploader': 'Deadline',
            'uploader_id': 'x1xm8ri',
            'age_limit': 0,
            'view_count': int,
        },
    }, {
        'url': 'https://www.dailymotion.com/video/x2iuewm_steam-machine-models-pricing-listed-on-steam-store-ign-news_videogames',
@ -141,7 +150,8 @@ class DailymotionIE(DailymotionBaseInfoExtractor):
        age_limit = self._rta_search(webpage)
-        description = self._og_search_description(webpage) or self._html_search_meta(
+        description = self._og_search_description(
            webpage, default=None) or self._html_search_meta(
            'description', webpage, 'description')
        view_count_str = self._search_regex(
@ -164,8 +174,34 @@ class DailymotionIE(DailymotionBaseInfoExtractor):
             r'__PLAYER_CONFIG__\s*=\s*({.+?});'],
            webpage, 'player v5', default=None)
        if player_v5:
-            player = self._parse_json(player_v5, video_id)
+            player = self._parse_json(player_v5, video_id, fatal=False) or {}
-            metadata = player['metadata']
+            metadata = try_get(player, lambda x: x['metadata'], dict)
            if not metadata:
                metadata_url = url_or_none(try_get(
                    player, lambda x: x['context']['metadata_template_url1']))
                if metadata_url:
                    metadata_url = metadata_url.replace(':videoId', video_id)
                else:
                    metadata_url = update_url_query(
                        'https://www.dailymotion.com/player/metadata/video/%s'
                        % video_id, {
                            'embedder': url,
                            'integration': 'inline',
                            'GK_PV5_NEON': '1',
                        })
                metadata = self._download_json(
                    metadata_url, video_id, 'Downloading metadata JSON')
            if try_get(metadata, lambda x: x['error']['type']) == 'password_protected':
                password = self._downloader.params.get('videopassword')
                if password:
                    r = int(metadata['id'][1:], 36)
                    us64e = lambda x: base64.urlsafe_b64encode(x).decode().strip('=')
                    t = ''.join(random.choice(string.ascii_letters) for i in range(10))
                    n = us64e(compat_struct_pack('I', r))
                    i = us64e(hashlib.md5(('%s%d%s' % (password, r, t)).encode()).digest())
                    metadata = self._download_json(
                        'http://www.dailymotion.com/player/metadata/video/p' + i + t + n, video_id)
            self._check_error(metadata)
@ -180,9 +216,12 @@ class DailymotionIE(DailymotionBaseInfoExtractor):
                        continue
                    ext = mimetype2ext(type_) or determine_ext(media_url)
                    if ext == 'm3u8':
-                        formats.extend(self._extract_m3u8_formats(
+                        m3u8_formats = self._extract_m3u8_formats(
                            media_url, video_id, 'mp4', preference=-1,
-                            m3u8_id='hls', fatal=False))
+                            m3u8_id='hls', fatal=False)
                        for f in m3u8_formats:
                            f['url'] = f['url'].split('#')[0]
                            formats.append(f)
                    elif ext == 'f4m':
                        formats.extend(self._extract_f4m_formats(
                            media_url, video_id, preference=-1, f4m_id='hds', fatal=False))
@ -299,8 +338,8 @@ class DailymotionIE(DailymotionBaseInfoExtractor):
    def _check_error(self, info):
        error = info.get('error')
-        if info.get('error') is not None:
+        if error:
-            title = error['title']
+            title = error.get('title') or error['message']
            # See https://developer.dailymotion.com/api#access-error
            if error.get('code') == 'DM007':
                self.raise_geo_restricted(msg=title)
@ -325,17 +364,93 @@ class DailymotionIE(DailymotionBaseInfoExtractor):
 class DailymotionPlaylistIE(DailymotionBaseInfoExtractor):
    IE_NAME = 'dailymotion:playlist'
-    _VALID_URL = r'(?:https?://)?(?:www\.)?dailymotion\.[a-z]{2,3}/playlist/(?P<id>[^/?#&]+)'
+    _VALID_URL = r'(?:https?://)?(?:www\.)?dailymotion\.[a-z]{2,3}/playlist/(?P<id>x[0-9a-z]+)'
    _MORE_PAGES_INDICATOR = r'(?s)<div class="pages[^"]*">.*?<a\s+class="[^"]*?icon-arrow_right[^"]*?"'
    _PAGE_TEMPLATE = 'https://www.dailymotion.com/playlist/%s/%s'
    _TESTS = [{
        'url': 'http://www.dailymotion.com/playlist/xv4bw_nqtv_sport/1#video=xl8v3q',
        'info_dict': {
            'title': 'SPORT',
-            'id': 'xv4bw_nqtv_sport',
+            'id': 'xv4bw',
        },
        'playlist_mincount': 20,
    }]
    _PAGE_SIZE = 100
    def _fetch_page(self, playlist_id, authorizaion, page):
        page += 1
        videos = self._download_json(
            'https://graphql.api.dailymotion.com',
            playlist_id, 'Downloading page %d' % page,
            data=json.dumps({
                'query': '''{
  collection(xid: "%s") {
    videos(first: %d, page: %d) {
      pageInfo {
        hasNextPage
        nextPage
      }
      edges {
        node {
          xid
          url
        }
      }
    }
  }
 }''' % (playlist_id, self._PAGE_SIZE, page)
            }).encode(), headers={
                'Authorization': authorizaion,
                'Origin': 'https://www.dailymotion.com',
            })['data']['collection']['videos']
        for edge in videos['edges']:
            node = edge['node']
            yield self.url_result(
                node['url'], DailymotionIE.ie_key(), node['xid'])
    def _real_extract(self, url):
        playlist_id = self._match_id(url)
        webpage = self._download_webpage(url, playlist_id)
        api = self._parse_json(self._search_regex(
            r'__PLAYER_CONFIG__\s*=\s*({.+?});',
            webpage, 'player config'), playlist_id)['context']['api']
        auth = self._download_json(
            api.get('auth_url', 'https://graphql.api.dailymotion.com/oauth/token'),
            playlist_id, data=urlencode_postdata({
                'client_id': api.get('client_id', 'f1a362d288c1b98099c7'),
                'client_secret': api.get('client_secret', 'eea605b96e01c796ff369935357eca920c5da4c5'),
                'grant_type': 'client_credentials',
            }))
        authorizaion = '%s %s' % (auth.get('token_type', 'Bearer'), auth['access_token'])
        entries = OnDemandPagedList(functools.partial(
            self._fetch_page, playlist_id, authorizaion), self._PAGE_SIZE)
        return self.playlist_result(
            entries, playlist_id,
            self._og_search_title(webpage))
 class DailymotionUserIE(DailymotionBaseInfoExtractor):
    IE_NAME = 'dailymotion:user'
    _VALID_URL = r'https?://(?:www\.)?dailymotion\.[a-z]{2,3}/(?!(?:embed|swf|#|video|playlist)/)(?:(?:old/)?user/)?(?P<user>[^/]+)'
    _MORE_PAGES_INDICATOR = r'(?s)<div class="pages[^"]*">.*?<a\s+class="[^"]*?icon-arrow_right[^"]*?"'
    _PAGE_TEMPLATE = 'http://www.dailymotion.com/user/%s/%s'
    _TESTS = [{
        'url': 'https://www.dailymotion.com/user/nqtv',
        'info_dict': {
            'id': 'nqtv',
            'title': 'Rémi Gaillard',
        },
        'playlist_mincount': 100,
    }, {
        'url': 'http://www.dailymotion.com/user/UnderProject',
        'info_dict': {
            'id': 'UnderProject',
            'title': 'UnderProject',
        },
        'playlist_mincount': 1800,
        'expected_warnings': [
            'Stopped at duplicated page',
        ],
        'skip': 'Takes too long time',
    }]
    def _extract_entries(self, id):
        video_ids = set()
@ -361,43 +476,6 @@ class DailymotionPlaylistIE(DailymotionBaseInfoExtractor):
            if re.search(self._MORE_PAGES_INDICATOR, webpage) is None:
                break
    def _real_extract(self, url):
        mobj = re.match(self._VALID_URL, url)
        playlist_id = mobj.group('id')
        webpage = self._download_webpage(url, playlist_id)
        return {
            '_type': 'playlist',
            'id': playlist_id,
            'title': self._og_search_title(webpage),
            'entries': self._extract_entries(playlist_id),
        }
 class DailymotionUserIE(DailymotionPlaylistIE):
    IE_NAME = 'dailymotion:user'
    _VALID_URL = r'https?://(?:www\.)?dailymotion\.[a-z]{2,3}/(?!(?:embed|swf|#|video|playlist)/)(?:(?:old/)?user/)?(?P<user>[^/]+)'
    _PAGE_TEMPLATE = 'http://www.dailymotion.com/user/%s/%s'
    _TESTS = [{
        'url': 'https://www.dailymotion.com/user/nqtv',
        'info_dict': {
            'id': 'nqtv',
            'title': 'Rémi Gaillard',
        },
        'playlist_mincount': 100,
    }, {
        'url': 'http://www.dailymotion.com/user/UnderProject',
        'info_dict': {
            'id': 'UnderProject',
            'title': 'UnderProject',
        },
        'playlist_mincount': 1800,
        'expected_warnings': [
            'Stopped at duplicated page',
        ],
        'skip': 'Takes too long time',
    }]
    def _real_extract(self, url):
        mobj = re.match(self._VALID_URL, url)
        user = mobj.group('user')
--- a/youtube_dl/extractor/dctp.py
+++ b/youtube_dl/extractor/dctp.py
@ -5,13 +5,16 @@ from .common import InfoExtractor
 from ..compat import compat_str
 from ..utils import (
    float_or_none,
-    unified_strdate,
+    int_or_none,
    unified_timestamp,
    url_or_none,
 )
 class DctpTvIE(InfoExtractor):
    _VALID_URL = r'https?://(?:www\.)?dctp\.tv/(?:#/)?filme/(?P<id>[^/?#&]+)'
-    _TEST = {
+    _TESTS = [{
        # 4x3
        'url': 'http://www.dctp.tv/filme/videoinstallation-fuer-eine-kaufhausfassade/',
        'info_dict': {
            'id': '95eaa4f33dad413aa17b4ee613cccc6c',
@ -19,37 +22,55 @@ class DctpTvIE(InfoExtractor):
            'ext': 'flv',
            'title': 'Videoinstallation für eine Kaufhausfassade',
            'description': 'Kurzfilm',
            'upload_date': '20110407',
            'thumbnail': r're:^https?://.*\.jpg$',
            'duration': 71.24,
            'timestamp': 1302172322,
            'upload_date': '20110407',
        },
        'params': {
            # rtmp download
            'skip_download': True,
        },
-    }
+    }, {
        # 16x9
        'url': 'http://www.dctp.tv/filme/sind-youtuber-die-besseren-lehrer/',
        'only_matching': True,
    }]
    _BASE_URL = 'http://dctp-ivms2-restapi.s3.amazonaws.com'
    def _real_extract(self, url):
        display_id = self._match_id(url)
-        webpage = self._download_webpage(url, display_id)
+        version = self._download_json(
            '%s/version.json' % self._BASE_URL, display_id,
            'Downloading version JSON')
-        video_id = self._html_search_meta(
+        restapi_base = '%s/%s/restapi' % (
-            'DC.identifier', webpage, 'video id',
+            self._BASE_URL, version['version_name'])
            default=None) or self._search_regex(
            r'id=["\']uuid[^>]+>([^<]+)<', webpage, 'video id')
-        title = self._og_search_title(webpage)
+        info = self._download_json(
            '%s/slugs/%s.json' % (restapi_base, display_id), display_id,
            'Downloading video info JSON')
        media = self._download_json(
            '%s/media/%s.json' % (restapi_base, compat_str(info['object_id'])),
            display_id, 'Downloading media JSON')
        uuid = media['uuid']
        title = media['title']
        ratio = '16x9' if media.get('is_wide') else '4x3'
        play_path = 'mp4:%s_dctp_0500_%s.m4v' % (uuid, ratio)
        servers = self._download_json(
            'http://www.dctp.tv/streaming_servers/', display_id,
-            note='Downloading server list', fatal=False)
+            note='Downloading server list JSON', fatal=False)
        if servers:
            endpoint = next(
                server['endpoint']
                for server in servers
-                if isinstance(server.get('endpoint'), compat_str) and
+                if url_or_none(server.get('endpoint')) and
                'cloudfront' in server['endpoint'])
        else:
            endpoint = 'rtmpe://s2pqqn4u96e4j8.cloudfront.net/cfx/st/'
@ -60,27 +81,35 @@ class DctpTvIE(InfoExtractor):
        formats = [{
            'url': endpoint,
            'app': app,
-            'play_path': 'mp4:%s_dctp_0500_4x3.m4v' % video_id,
+            'play_path': play_path,
            'page_url': url,
-            'player_url': 'http://svm-prod-dctptv-static.s3.amazonaws.com/dctptv-relaunch2012-109.swf',
+            'player_url': 'http://svm-prod-dctptv-static.s3.amazonaws.com/dctptv-relaunch2012-110.swf',
            'ext': 'flv',
        }]
-        description = self._html_search_meta('DC.description', webpage)
+        thumbnails = []
-        upload_date = unified_strdate(
+        images = media.get('images')
-            self._html_search_meta('DC.date.created', webpage))
+        if isinstance(images, list):
-        thumbnail = self._og_search_thumbnail(webpage)
+            for image in images:
-        duration = float_or_none(self._search_regex(
+                if not isinstance(image, dict):
-            r'id=["\']duration_in_ms[^+]>(\d+)', webpage, 'duration',
+                    continue
-            default=None), scale=1000)
+                image_url = url_or_none(image.get('url'))
                if not image_url:
                    continue
                thumbnails.append({
                    'url': image_url,
                    'width': int_or_none(image.get('width')),
                    'height': int_or_none(image.get('height')),
                })
        return {
-            'id': video_id,
+            'id': uuid,
            'title': title,
            'formats': formats,
            'display_id': display_id,
-            'description': description,
+            'title': title,
-            'upload_date': upload_date,
+            'alt_title': media.get('subtitle'),
-            'thumbnail': thumbnail,
+            'description': media.get('description') or media.get('teaser'),
-            'duration': duration,
+            'timestamp': unified_timestamp(media.get('created')),
            'duration': float_or_none(media.get('duration_in_ms'), scale=1000),
            'thumbnails': thumbnails,
            'formats': formats,
        }
--- a/youtube_dl/extractor/discovery.py
+++ b/youtube_dl/extractor/discovery.py
@ -5,7 +5,10 @@ import re
 import string
 from .discoverygo import DiscoveryGoBaseIE
-from ..compat import compat_str
+from ..compat import (
    compat_str,
    compat_urllib_parse_unquote,
 )
 from ..utils import (
    ExtractorError,
    try_get,
@ -55,15 +58,27 @@ class DiscoveryIE(DiscoveryGoBaseIE):
        video = next(cb for cb in content_blocks if cb.get('type') == 'video')['content']['items'][0]
        video_id = video['id']
-        access_token = self._download_json(
+        access_token = None
-            'https://www.%s.com/anonymous' % site, display_id, query={
+        cookies = self._get_cookies(url)
-                'authRel': 'authorization',
+
-                'client_id': try_get(
+        # prefer Affiliate Auth Token over Anonymous Auth Token
-                    react_data, lambda x: x['application']['apiClientId'],
+        auth_storage_cookie = cookies.get('eosAf') or cookies.get('eosAn')
-                    compat_str) or '3020a40c2356a645b4b4',
+        if auth_storage_cookie and auth_storage_cookie.value:
-                'nonce': ''.join([random.choice(string.ascii_letters) for _ in range(32)]),
+            auth_storage = self._parse_json(compat_urllib_parse_unquote(
-                'redirectUri': 'https://fusion.ddmcdn.com/app/mercury-sdk/180/redirectHandler.html?https://www.%s.com' % site,
+                compat_urllib_parse_unquote(auth_storage_cookie.value)),
-            })['access_token']
+                video_id, fatal=False) or {}
            access_token = auth_storage.get('a') or auth_storage.get('access_token')
        if not access_token:
            access_token = self._download_json(
                'https://www.%s.com/anonymous' % site, display_id, query={
                    'authRel': 'authorization',
                    'client_id': try_get(
                        react_data, lambda x: x['application']['apiClientId'],
                        compat_str) or '3020a40c2356a645b4b4',
                    'nonce': ''.join([random.choice(string.ascii_letters) for _ in range(32)]),
                    'redirectUri': 'https://fusion.ddmcdn.com/app/mercury-sdk/180/redirectHandler.html?https://www.%s.com' % site,
                })['access_token']
        try:
            stream = self._download_json(
@ -72,7 +87,7 @@ class DiscoveryIE(DiscoveryGoBaseIE):
                    'Authorization': 'Bearer ' + access_token,
                })
        except ExtractorError as e:
-            if isinstance(e.cause, compat_HTTPError) and e.cause.code == 403:
+            if isinstance(e.cause, compat_HTTPError) and e.cause.code in (401, 403):
                e_description = self._parse_json(
                    e.cause.read().decode(), display_id)['description']
                if 'resource not available for country' in e_description:
--- a/youtube_dl/extractor/discoverygo.py
+++ b/youtube_dl/extractor/discoverygo.py
@ -3,7 +3,6 @@ from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
 from ..compat import compat_str
 from ..utils import (
    determine_ext,
    extract_attributes,
@ -12,6 +11,7 @@ from ..utils import (
    parse_age_limit,
    remove_end,
    unescapeHTML,
    url_or_none,
 )
@ -69,9 +69,8 @@ class DiscoveryGoBaseIE(InfoExtractor):
        captions = stream.get('captions')
        if isinstance(captions, list):
            for caption in captions:
-                subtitle_url = caption.get('fileUrl')
+                subtitle_url = url_or_none(caption.get('fileUrl'))
-                if (not subtitle_url or not isinstance(subtitle_url, compat_str) or
+                if not subtitle_url or not subtitle_url.startswith('http'):
                        not subtitle_url.startswith('http')):
                    continue
                lang = caption.get('fileLang', 'en')
                ext = determine_ext(subtitle_url)
--- a/youtube_dl/extractor/discoverynetworks.py
+++ b/youtube_dl/extractor/discoverynetworks.py
@ -3,8 +3,8 @@ from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
 from .brightcove import BrightcoveLegacyIE
 from .dplay import DPlayIE
 from ..compat import (
    compat_parse_qs,
    compat_urlparse,
@ -12,8 +12,13 @@ from ..compat import (
 from ..utils import smuggle_url
-class DiscoveryNetworksDeIE(InfoExtractor):
+class DiscoveryNetworksDeIE(DPlayIE):
-    _VALID_URL = r'https?://(?:www\.)?(?:discovery|tlc|animalplanet|dmax)\.de/(?:.*#(?P<id>\d+)|(?:[^/]+/)*videos/(?P<title>[^/?#]+))'
+    _VALID_URL = r'''(?x)https?://(?:www\.)?(?P<site>discovery|tlc|animalplanet|dmax)\.de/
                        (?:
                           .*\#(?P<id>\d+)|
                           (?:[^/]+/)*videos/(?P<display_id>[^/?#]+)|
                           programme/(?P<programme>[^/]+)/video/(?P<alternate_id>[^/]+)
                        )'''
    _TESTS = [{
        'url': 'http://www.tlc.de/sendungen/breaking-amish/videos/#3235167922001',
@ -40,6 +45,14 @@ class DiscoveryNetworksDeIE(InfoExtractor):
    def _real_extract(self, url):
        mobj = re.match(self._VALID_URL, url)
        alternate_id = mobj.group('alternate_id')
        if alternate_id:
            self._initialize_geo_bypass({
                'countries': ['DE'],
            })
            return self._get_disco_api_info(
                url, '%s/%s' % (mobj.group('programme'), alternate_id),
                'sonic-eu1-prod.disco-api.com', mobj.group('site') + 'de')
        brightcove_id = mobj.group('id')
        if not brightcove_id:
            title = mobj.group('title')
--- a/youtube_dl/extractor/dplay.py
+++ b/youtube_dl/extractor/dplay.py
@ -21,6 +21,7 @@ from ..utils import (
    unified_strdate,
    unified_timestamp,
    update_url_query,
    urljoin,
    USER_AGENTS,
 )
@ -97,12 +98,83 @@ class DPlayIE(InfoExtractor):
        'only_matching': True,
    }]
    def _get_disco_api_info(self, url, display_id, disco_host, realm):
        disco_base = 'https://' + disco_host
        token = self._download_json(
            '%s/token' % disco_base, display_id, 'Downloading token',
            query={
                'realm': realm,
            })['data']['attributes']['token']
        headers = {
            'Referer': url,
            'Authorization': 'Bearer ' + token,
        }
        video = self._download_json(
            '%s/content/videos/%s' % (disco_base, display_id), display_id,
            headers=headers, query={
                'include': 'show'
            })
        video_id = video['data']['id']
        info = video['data']['attributes']
        title = info['name']
        formats = []
        for format_id, format_dict in self._download_json(
                '%s/playback/videoPlaybackInfo/%s' % (disco_base, video_id),
                display_id, headers=headers)['data']['attributes']['streaming'].items():
            if not isinstance(format_dict, dict):
                continue
            format_url = format_dict.get('url')
            if not format_url:
                continue
            ext = determine_ext(format_url)
            if format_id == 'dash' or ext == 'mpd':
                formats.extend(self._extract_mpd_formats(
                    format_url, display_id, mpd_id='dash', fatal=False))
            elif format_id == 'hls' or ext == 'm3u8':
                formats.extend(self._extract_m3u8_formats(
                    format_url, display_id, 'mp4',
                    entry_protocol='m3u8_native', m3u8_id='hls',
                    fatal=False))
            else:
                formats.append({
                    'url': format_url,
                    'format_id': format_id,
                })
        self._sort_formats(formats)
        series = None
        try:
            included = video.get('included')
            if isinstance(included, list):
                show = next(e for e in included if e.get('type') == 'show')
                series = try_get(
                    show, lambda x: x['attributes']['name'], compat_str)
        except StopIteration:
            pass
        return {
            'id': video_id,
            'display_id': display_id,
            'title': title,
            'description': info.get('description'),
            'duration': float_or_none(
                info.get('videoDuration'), scale=1000),
            'timestamp': unified_timestamp(info.get('publishStart')),
            'series': series,
            'season_number': int_or_none(info.get('seasonNumber')),
            'episode_number': int_or_none(info.get('episodeNumber')),
            'age_limit': int_or_none(info.get('minimum_age')),
            'formats': formats,
        }
    def _real_extract(self, url):
        mobj = re.match(self._VALID_URL, url)
        display_id = mobj.group('id')
        domain = mobj.group('domain')
-        self._initialize_geo_bypass([mobj.group('country').upper()])
+        self._initialize_geo_bypass({
            'countries': [mobj.group('country').upper()],
        })
        webpage = self._download_webpage(url, display_id)
@ -111,72 +183,8 @@ class DPlayIE(InfoExtractor):
        if not video_id:
            host = mobj.group('host')
-            disco_base = 'https://disco-api.%s' % host
+            return self._get_disco_api_info(
-            self._download_json(
+                url, display_id, 'disco-api.' + host, host.replace('.', ''))
                '%s/token' % disco_base, display_id, 'Downloading token',
                query={
                    'realm': host.replace('.', ''),
                })
            video = self._download_json(
                '%s/content/videos/%s' % (disco_base, display_id), display_id,
                headers={
                    'Referer': url,
                    'x-disco-client': 'WEB:UNKNOWN:dplay-client:0.0.1',
                }, query={
                    'include': 'show'
                })
            video_id = video['data']['id']
            info = video['data']['attributes']
            title = info['name']
            formats = []
            for format_id, format_dict in self._download_json(
                    '%s/playback/videoPlaybackInfo/%s' % (disco_base, video_id),
                    display_id)['data']['attributes']['streaming'].items():
                if not isinstance(format_dict, dict):
                    continue
                format_url = format_dict.get('url')
                if not format_url:
                    continue
                ext = determine_ext(format_url)
                if format_id == 'dash' or ext == 'mpd':
                    formats.extend(self._extract_mpd_formats(
                        format_url, display_id, mpd_id='dash', fatal=False))
                elif format_id == 'hls' or ext == 'm3u8':
                    formats.extend(self._extract_m3u8_formats(
                        format_url, display_id, 'mp4',
                        entry_protocol='m3u8_native', m3u8_id='hls',
                        fatal=False))
                else:
                    formats.append({
                        'url': format_url,
                        'format_id': format_id,
                    })
            self._sort_formats(formats)
            series = None
            try:
                included = video.get('included')
                if isinstance(included, list):
                    show = next(e for e in included if e.get('type') == 'show')
                    series = try_get(
                        show, lambda x: x['attributes']['name'], compat_str)
            except StopIteration:
                pass
            return {
                'id': video_id,
                'display_id': display_id,
                'title': title,
                'description': info.get('description'),
                'duration': float_or_none(
                    info.get('videoDuration'), scale=1000),
                'timestamp': unified_timestamp(info.get('publishStart')),
                'series': series,
                'season_number': int_or_none(info.get('seasonNumber')),
                'episode_number': int_or_none(info.get('episodeNumber')),
                'age_limit': int_or_none(info.get('minimum_age')),
                'formats': formats,
            }
        info = self._download_json(
            'http://%s/api/v2/ajax/videos?video_id=%s' % (domain, video_id),
@ -303,9 +311,11 @@ class DPlayItIE(InfoExtractor):
        if not info:
            info_url = self._search_regex(
-                r'url\s*[:=]\s*["\']((?:https?:)?//[^/]+/playback/videoPlaybackInfo/\d+)',
+                (r'playback_json_url\s*:\s*(["\'])(?P<url>(?:(?!\1).)+)\1',
-                webpage, 'info url')
+                 r'url\s*[:=]\s*["\'](?P<url>(?:https?:)?//[^/]+/playback/videoPlaybackInfo/\d+)'),
                webpage, 'info url', group='url')
            info_url = urljoin(url, info_url)
            video_id = info_url.rpartition('/')[-1]
            try:
@ -315,6 +325,8 @@ class DPlayItIE(InfoExtractor):
                            'dplayit_token').value,
                        'Referer': url,
                    })
                if isinstance(info, compat_str):
                    info = self._parse_json(info, display_id)
            except ExtractorError as e:
                if isinstance(e.cause, compat_HTTPError) and e.cause.code in (400, 403):
                    info = self._parse_json(e.cause.read().decode('utf-8'), display_id)
@ -330,6 +342,7 @@ class DPlayItIE(InfoExtractor):
        formats = self._extract_m3u8_formats(
            hls_url, display_id, ext='mp4', entry_protocol='m3u8_native',
            m3u8_id='hls')
        self._sort_formats(formats)
        series = self._html_search_regex(
            r'(?s)<h1[^>]+class=["\'].*?\bshow_title\b.*?["\'][^>]*>(.+?)</h1>',
--- a/youtube_dl/extractor/dramafever.py
+++ b/youtube_dl/extractor/dramafever.py
@ -2,26 +2,26 @@
 from __future__ import unicode_literals
 import itertools
 import json
-from .amp import AMPIE
+from .common import InfoExtractor
 from ..compat import (
    compat_HTTPError,
    compat_urlparse,
 )
 from ..utils import (
    ExtractorError,
    clean_html,
    ExtractorError,
    int_or_none,
-    remove_end,
+    parse_age_limit,
-    sanitized_Request,
+    parse_duration,
-    urlencode_postdata
+    unified_timestamp,
    url_or_none,
 )
-class DramaFeverBaseIE(AMPIE):
+class DramaFeverBaseIE(InfoExtractor):
    _LOGIN_URL = 'https://www.dramafever.com/accounts/login/'
    _NETRC_MACHINE = 'dramafever'
    _GEO_COUNTRIES = ['US', 'CA']
    _CONSUMER_SECRET = 'DA59dtVXYLxajktV'
@ -38,11 +38,11 @@ class DramaFeverBaseIE(AMPIE):
            'consumer secret', default=self._CONSUMER_SECRET)
    def _real_initialize(self):
        self._login()
        self._consumer_secret = self._get_consumer_secret()
        self._login()
    def _login(self):
-        (username, password) = self._get_login_info()
+        username, password = self._get_login_info()
        if username is None:
            return
@ -51,37 +51,49 @@ class DramaFeverBaseIE(AMPIE):
            'password': password,
        }
-        request = sanitized_Request(
+        try:
-            self._LOGIN_URL, urlencode_postdata(login_form))
+            response = self._download_json(
-        response = self._download_webpage(
+                'https://www.dramafever.com/api/users/login', None, 'Logging in',
-            request, None, 'Logging in')
+                data=json.dumps(login_form).encode('utf-8'), headers={
                    'x-consumer-key': self._consumer_secret,
                })
        except ExtractorError as e:
            if isinstance(e.cause, compat_HTTPError) and e.cause.code in (403, 404):
                response = self._parse_json(
                    e.cause.read().decode('utf-8'), None)
            else:
                raise
-        if all(logout_pattern not in response
+        # Successful login
-               for logout_pattern in ['href="/accounts/logout/"', '>Log out<']):
+        if response.get('result') or response.get('guid') or response.get('user_guid'):
-            error = self._html_search_regex(
+            return
-                r'(?s)<h\d[^>]+\bclass="hidden-xs prompt"[^>]*>(.+?)</h\d',
+
-                response, 'error message', default=None)
+        errors = response.get('errors')
-            if error:
+        if errors and isinstance(errors, list):
-                raise ExtractorError('Unable to login: %s' % error, expected=True)
+            error = errors[0]
-            raise ExtractorError('Unable to log in')
+            message = error.get('message') or error['reason']
            raise ExtractorError('Unable to login: %s' % message, expected=True)
        raise ExtractorError('Unable to log in')
 class DramaFeverIE(DramaFeverBaseIE):
    IE_NAME = 'dramafever'
    _VALID_URL = r'https?://(?:www\.)?dramafever\.com/(?:[^/]+/)?drama/(?P<id>[0-9]+/[0-9]+)(?:/|$)'
    _TESTS = [{
-        'url': 'http://www.dramafever.com/drama/4512/1/Cooking_with_Shin/',
+        'url': 'https://www.dramafever.com/drama/4274/1/Heirs/',
        'info_dict': {
-            'id': '4512.1',
+            'id': '4274.1',
-            'ext': 'flv',
+            'ext': 'wvm',
-            'title': 'Cooking with Shin',
+            'title': 'Heirs - Episode 1',
-            'description': 'md5:a8eec7942e1664a6896fcd5e1287bfd0',
+            'description': 'md5:362a24ba18209f6276e032a651c50bc2',
            'thumbnail': r're:^https?://.*\.jpg',
            'duration': 3783,
            'timestamp': 1381354993,
            'upload_date': '20131009',
            'series': 'Heirs',
            'season_number': 1,
            'episode': 'Episode 1',
            'episode_number': 1,
            'thumbnail': r're:^https?://.*\.jpg',
            'timestamp': 1404336058,
            'upload_date': '20140702',
            'duration': 344,
        },
        'params': {
            # m3u8 download
@ -110,50 +122,95 @@ class DramaFeverIE(DramaFeverBaseIE):
        'only_matching': True,
    }]
    def _call_api(self, path, video_id, note, fatal=False):
        return self._download_json(
            'https://www.dramafever.com/api/5/' + path,
            video_id, note=note, headers={
                'x-consumer-key': self._consumer_secret,
            }, fatal=fatal)
    def _get_subtitles(self, video_id):
        subtitles = {}
        subs = self._call_api(
            'video/%s/subtitles/webvtt/' % video_id, video_id,
            'Downloading subtitles JSON', fatal=False)
        if not subs or not isinstance(subs, list):
            return subtitles
        for sub in subs:
            if not isinstance(sub, dict):
                continue
            sub_url = url_or_none(sub.get('url'))
            if not sub_url:
                continue
            subtitles.setdefault(
                sub.get('code') or sub.get('language') or 'en', []).append({
                    'url': sub_url
                })
        return subtitles
    def _real_extract(self, url):
        video_id = self._match_id(url).replace('/', '.')
        try:
            info = self._extract_feed_info(
                'http://www.dramafever.com/amp/episode/feed.json?guid=%s' % video_id)
        except ExtractorError as e:
            if isinstance(e.cause, compat_HTTPError):
                self.raise_geo_restricted(
                    msg='Currently unavailable in your country',
                    countries=self._GEO_COUNTRIES)
            raise
        # title is postfixed with video id for some reason, removing
        if info.get('title'):
            info['title'] = remove_end(info['title'], video_id).strip()
        series_id, episode_number = video_id.split('.')
        episode_info = self._download_json(
            # We only need a single episode info, so restricting page size to one episode
            # and dealing with page number as with episode number
            r'http://www.dramafever.com/api/4/episode/series/?cs=%s&series_id=%s&page_number=%s&page_size=1'
            % (self._consumer_secret, series_id, episode_number),
            video_id, 'Downloading episode info JSON', fatal=False)
        if episode_info:
            value = episode_info.get('value')
            if isinstance(value, list):
                for v in value:
                    if v.get('type') == 'Episode':
                        subfile = v.get('subfile') or v.get('new_subfile')
                        if subfile and subfile != 'http://www.dramafever.com/st/':
                            info.setdefault('subtitles', {}).setdefault('English', []).append({
                                'ext': 'srt',
                                'url': subfile,
                            })
                        episode_number = int_or_none(v.get('number'))
                        episode_fallback = 'Episode'
                        if episode_number:
                            episode_fallback += ' %d' % episode_number
                        info['episode'] = v.get('title') or episode_fallback
                        info['episode_number'] = episode_number
                        break
-        return info
+        video = self._call_api(
            'series/%s/episodes/%s/' % (series_id, episode_number), video_id,
            'Downloading video JSON')
        formats = []
        download_assets = video.get('download_assets')
        if download_assets and isinstance(download_assets, dict):
            for format_id, format_dict in download_assets.items():
                if not isinstance(format_dict, dict):
                    continue
                format_url = url_or_none(format_dict.get('url'))
                if not format_url:
                    continue
                formats.append({
                    'url': format_url,
                    'format_id': format_id,
                    'filesize': int_or_none(video.get('filesize')),
                })
        stream = self._call_api(
            'video/%s/stream/' % video_id, video_id, 'Downloading stream JSON',
            fatal=False)
        if stream:
            stream_url = stream.get('stream_url')
            if stream_url:
                formats.extend(self._extract_m3u8_formats(
                    stream_url, video_id, 'mp4', entry_protocol='m3u8_native',
                    m3u8_id='hls', fatal=False))
        self._sort_formats(formats)
        title = video.get('title') or 'Episode %s' % episode_number
        description = video.get('description')
        thumbnail = video.get('thumbnail')
        timestamp = unified_timestamp(video.get('release_date'))
        duration = parse_duration(video.get('duration'))
        age_limit = parse_age_limit(video.get('tv_rating'))
        series = video.get('series_title')
        season_number = int_or_none(video.get('season'))
        if series:
            title = '%s - %s' % (series, title)
        subtitles = self.extract_subtitles(video_id)
        return {
            'id': video_id,
            'title': title,
            'description': description,
            'thumbnail': thumbnail,
            'duration': duration,
            'timestamp': timestamp,
            'age_limit': age_limit,
            'series': series,
            'season_number': season_number,
            'episode_number': int_or_none(episode_number),
            'formats': formats,
            'subtitles': subtitles,
        }
 class DramaFeverSeriesIE(DramaFeverBaseIE):
--- a/youtube_dl/extractor/dreisat.py
+++ b/youtube_dl/extractor/dreisat.py
@ -8,7 +8,6 @@ from ..utils import (
    unified_strdate,
    xpath_text,
    determine_ext,
    qualities,
    float_or_none,
    ExtractorError,
 )
@ -16,7 +15,8 @@ from ..utils import (
 class DreiSatIE(InfoExtractor):
    IE_NAME = '3sat'
-    _VALID_URL = r'(?:https?://)?(?:www\.)?3sat\.de/mediathek/(?:index\.php|mediathek\.php)?\?(?:(?:mode|display)=[^&]+&)*obj=(?P<id>[0-9]+)$'
+    _GEO_COUNTRIES = ['DE']
    _VALID_URL = r'https?://(?:www\.)?3sat\.de/mediathek/(?:(?:index|mediathek)\.php)?\?(?:(?:mode|display)=[^&]+&)*obj=(?P<id>[0-9]+)'
    _TESTS = [
        {
            'url': 'http://www.3sat.de/mediathek/index.php?mode=play&obj=45918',
@ -43,7 +43,8 @@ class DreiSatIE(InfoExtractor):
    def _parse_smil_formats(self, smil, smil_url, video_id, namespace=None, f4m_params=None, transform_rtmp_url=None):
        param_groups = {}
        for param_group in smil.findall(self._xpath_ns('./head/paramGroup', namespace)):
-            group_id = param_group.attrib.get(self._xpath_ns('id', 'http://www.w3.org/XML/1998/namespace'))
+            group_id = param_group.get(self._xpath_ns(
                'id', 'http://www.w3.org/XML/1998/namespace'))
            params = {}
            for param in param_group:
                params[param.get('name')] = param.get('value')
@ -54,7 +55,7 @@ class DreiSatIE(InfoExtractor):
            src = video.get('src')
            if not src:
                continue
-            bitrate = float_or_none(video.get('system-bitrate') or video.get('systemBitrate'), 1000)
+            bitrate = int_or_none(self._search_regex(r'_(\d+)k', src, 'bitrate', None)) or float_or_none(video.get('system-bitrate') or video.get('systemBitrate'), 1000)
            group_id = video.get('paramGroup')
            param_group = param_groups[group_id]
            for proto in param_group['protocols'].split(','):
@ -75,66 +76,36 @@ class DreiSatIE(InfoExtractor):
            note='Downloading video info',
            errnote='Failed to download video info')
-        status_code = doc.find('./status/statuscode')
+        status_code = xpath_text(doc, './status/statuscode')
-        if status_code is not None and status_code.text != 'ok':
+        if status_code and status_code != 'ok':
-            code = status_code.text
+            if status_code == 'notVisibleAnymore':
            if code == 'notVisibleAnymore':
                message = 'Video %s is not available' % video_id
            else:
-                message = '%s returned error: %s' % (self.IE_NAME, code)
+                message = '%s returned error: %s' % (self.IE_NAME, status_code)
            raise ExtractorError(message, expected=True)
-        title = doc.find('.//information/title').text
+        title = xpath_text(doc, './/information/title', 'title', True)
        description = xpath_text(doc, './/information/detail', 'description')
        duration = int_or_none(xpath_text(doc, './/details/lengthSec', 'duration'))
        uploader = xpath_text(doc, './/details/originChannelTitle', 'uploader')
        uploader_id = xpath_text(doc, './/details/originChannelId', 'uploader id')
        upload_date = unified_strdate(xpath_text(doc, './/details/airtime', 'upload date'))
-        def xml_to_thumbnails(fnode):
+        urls = []
            thumbnails = []
            for node in fnode:
                thumbnail_url = node.text
                if not thumbnail_url:
                    continue
                thumbnail = {
                    'url': thumbnail_url,
                }
                if 'key' in node.attrib:
                    m = re.match('^([0-9]+)x([0-9]+)$', node.attrib['key'])
                    if m:
                        thumbnail['width'] = int(m.group(1))
                        thumbnail['height'] = int(m.group(2))
                thumbnails.append(thumbnail)
            return thumbnails
        thumbnails = xml_to_thumbnails(doc.findall('.//teaserimages/teaserimage'))
        format_nodes = doc.findall('.//formitaeten/formitaet')
        quality = qualities(['veryhigh', 'high', 'med', 'low'])
        def get_quality(elem):
            return quality(xpath_text(elem, 'quality'))
        format_nodes.sort(key=get_quality)
        format_ids = []
        formats = []
-        for fnode in format_nodes:
+        for fnode in doc.findall('.//formitaeten/formitaet'):
-            video_url = fnode.find('url').text
+            video_url = xpath_text(fnode, 'url')
-            is_available = 'http://www.metafilegenerator' not in video_url
+            if not video_url or video_url in urls:
            if not is_available:
                continue
            urls.append(video_url)
            is_available = 'http://www.metafilegenerator' not in video_url
            geoloced = 'static_geoloced_online' in video_url
            if not is_available or geoloced:
                continue
            format_id = fnode.attrib['basetype']
            quality = xpath_text(fnode, './quality', 'quality')
            format_m = re.match(r'''(?x)
                (?P<vcodec>[^_]+)_(?P<acodec>[^_]+)_(?P<container>[^_]+)_
                (?P<proto>[^_]+)_(?P<index>[^_]+)_(?P<indexproto>[^_]+)
            ''', format_id)
            ext = determine_ext(video_url, None) or format_m.group('container')
            if ext not in ('smil', 'f4m', 'm3u8'):
                format_id = format_id + '-' + quality
            if format_id in format_ids:
                continue
            if ext == 'meta':
                continue
@ -147,24 +118,23 @@ class DreiSatIE(InfoExtractor):
                if video_url.startswith('https://'):
                    continue
                formats.extend(self._extract_m3u8_formats(
-                    video_url, video_id, 'mp4', m3u8_id=format_id, fatal=False))
+                    video_url, video_id, 'mp4', 'm3u8_native',
                    m3u8_id=format_id, fatal=False))
            elif ext == 'f4m':
                formats.extend(self._extract_f4m_formats(
                    video_url, video_id, f4m_id=format_id, fatal=False))
            else:
-                proto = format_m.group('proto').lower()
+                quality = xpath_text(fnode, './quality')
                if quality:
                    format_id += '-' + quality
-                abr = int_or_none(xpath_text(fnode, './audioBitrate', 'abr'), 1000)
+                abr = int_or_none(xpath_text(fnode, './audioBitrate'), 1000)
-                vbr = int_or_none(xpath_text(fnode, './videoBitrate', 'vbr'), 1000)
+                vbr = int_or_none(xpath_text(fnode, './videoBitrate'), 1000)
-                width = int_or_none(xpath_text(fnode, './width', 'width'))
+                tbr = int_or_none(self._search_regex(
-                height = int_or_none(xpath_text(fnode, './height', 'height'))
+                    r'_(\d+)k', video_url, 'bitrate', None))
-
+                if tbr and vbr and not abr:
-                filesize = int_or_none(xpath_text(fnode, './filesize', 'filesize'))
+                    abr = tbr - vbr
                format_note = ''
                if not format_note:
                    format_note = None
                formats.append({
                    'format_id': format_id,
@ -174,31 +144,50 @@ class DreiSatIE(InfoExtractor):
                    'vcodec': format_m.group('vcodec'),
                    'abr': abr,
                    'vbr': vbr,
-                    'width': width,
+                    'tbr': tbr,
-                    'height': height,
+                    'width': int_or_none(xpath_text(fnode, './width')),
-                    'filesize': filesize,
+                    'height': int_or_none(xpath_text(fnode, './height')),
-                    'format_note': format_note,
+                    'filesize': int_or_none(xpath_text(fnode, './filesize')),
-                    'protocol': proto,
+                    'protocol': format_m.group('proto').lower(),
                    '_available': is_available,
                })
-            format_ids.append(format_id)
+
        geolocation = xpath_text(doc, './/details/geolocation')
        if not formats and geolocation and geolocation != 'none':
            self.raise_geo_restricted(countries=self._GEO_COUNTRIES)
        self._sort_formats(formats)
        thumbnails = []
        for node in doc.findall('.//teaserimages/teaserimage'):
            thumbnail_url = node.text
            if not thumbnail_url:
                continue
            thumbnail = {
                'url': thumbnail_url,
            }
            thumbnail_key = node.get('key')
            if thumbnail_key:
                m = re.match('^([0-9]+)x([0-9]+)$', thumbnail_key)
                if m:
                    thumbnail['width'] = int(m.group(1))
                    thumbnail['height'] = int(m.group(2))
            thumbnails.append(thumbnail)
        upload_date = unified_strdate(xpath_text(doc, './/details/airtime'))
        return {
            'id': video_id,
            'title': title,
-            'description': description,
+            'description': xpath_text(doc, './/information/detail'),
-            'duration': duration,
+            'duration': int_or_none(xpath_text(doc, './/details/lengthSec')),
            'thumbnails': thumbnails,
-            'uploader': uploader,
+            'uploader': xpath_text(doc, './/details/originChannelTitle'),
-            'uploader_id': uploader_id,
+            'uploader_id': xpath_text(doc, './/details/originChannelId'),
            'upload_date': upload_date,
            'formats': formats,
        }
    def _real_extract(self, url):
-        mobj = re.match(self._VALID_URL, url)
+        video_id = self._match_id(url)
-        video_id = mobj.group('id')
+        details_url = 'http://www.3sat.de/mediathek/xmlservice/web/beitragsDetails?id=%s' % video_id
        details_url = 'http://www.3sat.de/mediathek/xmlservice/web/beitragsDetails?ak=web&id=%s' % video_id
        return self.extract_from_xml_url(video_id, details_url)
--- a/youtube_dl/extractor/drtuber.py
+++ b/youtube_dl/extractor/drtuber.py
@ -66,7 +66,9 @@ class DrTuberIE(InfoExtractor):
        self._sort_formats(formats)
        title = self._html_search_regex(
-            (r'class="title_watch"[^>]*><(?:p|h\d+)[^>]*>([^<]+)<',
+            (r'<h1[^>]+class=["\']title[^>]+>([^<]+)',
             r'<title>([^<]+)\s*@\s+DrTuber',
             r'class="title_watch"[^>]*><(?:p|h\d+)[^>]*>([^<]+)<',
             r'<p[^>]+class="title_substrate">([^<]+)</p>',
             r'<title>([^<]+) - \d+'),
            webpage, 'title')
--- a/youtube_dl/extractor/dtube.py
+++ b/youtube_dl/extractor/dtube.py
@ -0,0 +1,83 @@
 # coding: utf-8
 from __future__ import unicode_literals
 import json
 import re
 from socket import timeout
 from .common import InfoExtractor
 from ..utils import (
    int_or_none,
    parse_iso8601,
 )
 class DTubeIE(InfoExtractor):
    _VALID_URL = r'https?://(?:www\.)?d\.tube/(?:#!/)?v/(?P<uploader_id>[0-9a-z.-]+)/(?P<id>[0-9a-z]{8})'
    _TEST = {
        'url': 'https://d.tube/#!/v/benswann/zqd630em',
        'md5': 'a03eaa186618ffa7a3145945543a251e',
        'info_dict': {
            'id': 'zqd630em',
            'ext': 'mp4',
            'title': 'Reality Check: FDA\'s Disinformation Campaign on Kratom',
            'description': 'md5:700d164e066b87f9eac057949e4227c2',
            'uploader_id': 'benswann',
            'upload_date': '20180222',
            'timestamp': 1519328958,
        },
        'params': {
            'format': '480p',
        },
    }
    def _real_extract(self, url):
        uploader_id, video_id = re.match(self._VALID_URL, url).groups()
        result = self._download_json('https://api.steemit.com/', video_id, data=json.dumps({
            'jsonrpc': '2.0',
            'method': 'get_content',
            'params': [uploader_id, video_id],
        }).encode())['result']
        metadata = json.loads(result['json_metadata'])
        video = metadata['video']
        content = video['content']
        info = video.get('info', {})
        title = info.get('title') or result['title']
        def canonical_url(h):
            if not h:
                return None
            return 'https://ipfs.io/ipfs/' + h
        formats = []
        for q in ('240', '480', '720', '1080', ''):
            video_url = canonical_url(content.get('video%shash' % q))
            if not video_url:
                continue
            format_id = (q + 'p') if q else 'Source'
            try:
                self.to_screen('%s: Checking %s video format URL' % (video_id, format_id))
                self._downloader._opener.open(video_url, timeout=5).close()
            except timeout:
                self.to_screen(
                    '%s: %s URL is invalid, skipping' % (video_id, format_id))
                continue
            formats.append({
                'format_id': format_id,
                'url': video_url,
                'height': int_or_none(q),
                'ext': 'mp4',
            })
        return {
            'id': video_id,
            'title': title,
            'description': content.get('description'),
            'thumbnail': canonical_url(info.get('snaphash')),
            'tags': content.get('tags') or metadata.get('tags'),
            'duration': info.get('duration'),
            'formats': formats,
            'timestamp': parse_iso8601(result.get('created')),
            'uploader_id': uploader_id,
        }
--- a/youtube_dl/extractor/dvtv.py
+++ b/youtube_dl/extractor/dvtv.py
@ -91,17 +91,6 @@ class DVTVIE(InfoExtractor):
    }, {
        'url': 'http://video.aktualne.cz/v-cechach-poprve-zazni-zelenkova-zrestaurovana-mse/r~45b4b00483ec11e4883b002590604f2e/',
        'only_matching': True,
    }, {
        'url': 'https://video.aktualne.cz/dvtv/babis-a-zeman-nesou-vinu-za-to-ze-nemame-jasno-v-tom-kdo-bud/r~026afb54fad711e79704ac1f6b220ee8/',
        'md5': '87defe16681b1429c91f7a74809823c6',
        'info_dict': {
            'id': 'f5ae72f6fad611e794dbac1f6b220ee8',
            'ext': 'mp4',
            'title': 'Babiš a Zeman nesou vinu za to, že nemáme jasno v tom, kdo bude vládnout, říká Pekarová Adamová',
        },
        'params': {
            'skip_download': True,
        },
    }]
    def _parse_video_metadata(self, js, video_id, live_js=None):
--- a/youtube_dl/extractor/eagleplatform.py
+++ b/youtube_dl/extractor/eagleplatform.py
@ -4,14 +4,12 @@ from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
-from ..compat import (
+from ..compat import compat_HTTPError
    compat_HTTPError,
    compat_str,
 )
 from ..utils import (
    ExtractorError,
    int_or_none,
    unsmuggle_url,
    url_or_none,
 )
@ -177,7 +175,7 @@ class EaglePlatformIE(InfoExtractor):
            video_id, 'Downloading mp4 JSON', fatal=False)
        if mp4_data:
            for format_id, format_url in mp4_data.get('data', {}).items():
-                if not isinstance(format_url, compat_str):
+                if not url_or_none(format_url):
                    continue
                height = int_or_none(format_id)
                if height is not None and m3u8_formats_dict.get(height):
--- a/youtube_dl/extractor/egghead.py
+++ b/youtube_dl/extractor/egghead.py
@ -8,6 +8,7 @@ from ..utils import (
    int_or_none,
    try_get,
    unified_timestamp,
    url_or_none,
 )
@ -34,8 +35,8 @@ class EggheadCourseIE(InfoExtractor):
        entries = []
        for lesson in lessons:
-            lesson_url = lesson.get('http_url')
+            lesson_url = url_or_none(lesson.get('http_url'))
-            if not lesson_url or not isinstance(lesson_url, compat_str):
+            if not lesson_url:
                continue
            lesson_id = lesson.get('id')
            if lesson_id:
@ -95,7 +96,8 @@ class EggheadLessonIE(InfoExtractor):
        formats = []
        for _, format_url in lesson['media_urls'].items():
-            if not format_url or not isinstance(format_url, compat_str):
+            format_url = url_or_none(format_url)
            if not format_url:
                continue
            ext = determine_ext(format_url)
            if ext == 'm3u8':
--- a/youtube_dl/extractor/eporner.py
+++ b/youtube_dl/extractor/eporner.py
@ -9,8 +9,10 @@ from ..utils import (
    encode_base_n,
    ExtractorError,
    int_or_none,
    merge_dicts,
    parse_duration,
    str_to_int,
    url_or_none,
 )
@ -24,10 +26,16 @@ class EpornerIE(InfoExtractor):
            'display_id': 'Infamous-Tiffany-Teen-Strip-Tease-Video',
            'ext': 'mp4',
            'title': 'Infamous Tiffany Teen Strip Tease Video',
            'description': 'md5:764f39abf932daafa37485eb46efa152',
            'timestamp': 1232520922,
            'upload_date': '20090121',
            'duration': 1838,
            'view_count': int,
            'age_limit': 18,
        },
        'params': {
            'proxy': '127.0.0.1:8118'
        }
    }, {
        # New (May 2016) URL layout
        'url': 'http://www.eporner.com/hd-porn/3YRUtzMcWn0/Star-Wars-XXX-Parody/',
@ -82,8 +90,8 @@ class EpornerIE(InfoExtractor):
            for format_id, format_dict in formats_dict.items():
                if not isinstance(format_dict, dict):
                    continue
-                src = format_dict.get('src')
+                src = url_or_none(format_dict.get('src'))
-                if not isinstance(src, compat_str) or not src.startswith('http'):
+                if not src or not src.startswith('http'):
                    continue
                if kind == 'hls':
                    formats.extend(self._extract_m3u8_formats(
@ -103,12 +111,15 @@ class EpornerIE(InfoExtractor):
                    })
        self._sort_formats(formats)
-        duration = parse_duration(self._html_search_meta('duration', webpage))
+        json_ld = self._search_json_ld(webpage, display_id, default={})
        duration = parse_duration(self._html_search_meta(
            'duration', webpage, default=None))
        view_count = str_to_int(self._search_regex(
            r'id="cinemaviews">\s*([0-9,]+)\s*<small>views',
            webpage, 'view count', fatal=False))
-        return {
+        return merge_dicts(json_ld, {
            'id': video_id,
            'display_id': display_id,
            'title': title,
@ -116,4 +127,4 @@ class EpornerIE(InfoExtractor):
            'view_count': view_count,
            'formats': formats,
            'age_limit': 18,
-        }
+        })
--- a/youtube_dl/extractor/etonline.py
+++ b/youtube_dl/extractor/etonline.py
@ -1,39 +0,0 @@
 # coding: utf-8
 from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
 class ETOnlineIE(InfoExtractor):
    _VALID_URL = r'https?://(?:www\.)?etonline\.com/(?:[^/]+/)*(?P<id>[^/?#&]+)'
    _TESTS = [{
        'url': 'http://www.etonline.com/tv/211130_dove_cameron_liv_and_maddie_emotional_episode_series_finale/',
        'info_dict': {
            'id': '211130_dove_cameron_liv_and_maddie_emotional_episode_series_finale',
            'title': 'md5:a21ec7d3872ed98335cbd2a046f34ee6',
            'description': 'md5:8b94484063f463cca709617c79618ccd',
        },
        'playlist_count': 2,
    }, {
        'url': 'http://www.etonline.com/media/video/here_are_the_stars_who_love_bringing_their_moms_as_dates_to_the_oscars-211359/',
        'only_matching': True,
    }]
    BRIGHTCOVE_URL_TEMPLATE = 'http://players.brightcove.net/1242911076001/default_default/index.html?videoId=ref:%s'
    def _real_extract(self, url):
        playlist_id = self._match_id(url)
        webpage = self._download_webpage(url, playlist_id)
        entries = [
            self.url_result(
                self.BRIGHTCOVE_URL_TEMPLATE % video_id, 'BrightcoveNew', video_id)
            for video_id in re.findall(
                r'site\.brightcove\s*\([^,]+,\s*["\'](title_\d+)', webpage)]
        return self.playlist_result(
            entries, playlist_id,
            self._og_search_title(webpage, fatal=False),
            self._og_search_description(webpage))
--- a/youtube_dl/extractor/expressen.py
+++ b/youtube_dl/extractor/expressen.py
@ -0,0 +1,98 @@
 # coding: utf-8
 from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
 from ..utils import (
    determine_ext,
    int_or_none,
    unescapeHTML,
    unified_timestamp,
 )
 class ExpressenIE(InfoExtractor):
    _VALID_URL = r'''(?x)
                    https?://
                        (?:www\.)?expressen\.se/
                        (?:(?:tvspelare/video|videoplayer/embed)/)?
                        tv/(?:[^/]+/)*
                        (?P<id>[^/?#&]+)
                    '''
    _TESTS = [{
        'url': 'https://www.expressen.se/tv/ledare/ledarsnack/ledarsnack-om-arbetslosheten-bland-kvinnor-i-speciellt-utsatta-omraden/',
        'md5': '2fbbe3ca14392a6b1b36941858d33a45',
        'info_dict': {
            'id': '8690962',
            'ext': 'mp4',
            'title': 'Ledarsnack: Om arbetslösheten bland kvinnor i speciellt utsatta områden',
            'description': 'md5:f38c81ff69f3de4d269bbda012fcbbba',
            'thumbnail': r're:^https?://.*\.jpg$',
            'duration': 788,
            'timestamp': 1526639109,
            'upload_date': '20180518',
        },
    }, {
        'url': 'https://www.expressen.se/tv/kultur/kulturdebatt-med-expressens-karin-olsson/',
        'only_matching': True,
    }, {
        'url': 'https://www.expressen.se/tvspelare/video/tv/ditv/ekonomistudion/experterna-har-ar-fragorna-som-avgor-valet/?embed=true&external=true&autoplay=true&startVolume=0&partnerId=di',
        'only_matching': True,
    }, {
        'url': 'https://www.expressen.se/videoplayer/embed/tv/ditv/ekonomistudion/experterna-har-ar-fragorna-som-avgor-valet/?embed=true&external=true&autoplay=true&startVolume=0&partnerId=di',
        'only_matching': True,
    }]
    @staticmethod
    def _extract_urls(webpage):
        return [
            mobj.group('url') for mobj in re.finditer(
                r'<iframe[^>]+\bsrc=(["\'])(?P<url>(?:https?:)?//(?:www\.)?expressen\.se/(?:tvspelare/video|videoplayer/embed)/tv/.+?)\1',
                webpage)]
    def _real_extract(self, url):
        display_id = self._match_id(url)
        webpage = self._download_webpage(url, display_id)
        def extract_data(name):
            return self._parse_json(
                self._search_regex(
                    r'data-%s=(["\'])(?P<value>(?:(?!\1).)+)\1' % name,
                    webpage, 'info', group='value'),
                display_id, transform_source=unescapeHTML)
        info = extract_data('video-tracking-info')
        video_id = info['videoId']
        data = extract_data('article-data')
        stream = data['stream']
        if determine_ext(stream) == 'm3u8':
            formats = self._extract_m3u8_formats(
                stream, display_id, 'mp4', entry_protocol='m3u8_native',
                m3u8_id='hls')
        else:
            formats = [{
                'url': stream,
            }]
        self._sort_formats(formats)
        title = info.get('titleRaw') or data['title']
        description = info.get('descriptionRaw')
        thumbnail = info.get('socialMediaImage') or data.get('image')
        duration = int_or_none(info.get('videoTotalSecondsDuration') or
                               data.get('totalSecondsDuration'))
        timestamp = unified_timestamp(info.get('publishDate'))
        return {
            'id': video_id,
            'display_id': display_id,
            'title': title,
            'description': description,
            'thumbnail': thumbnail,
            'duration': duration,
            'timestamp': timestamp,
            'formats': formats,
        }
--- a/youtube_dl/extractor/extractors.py
+++ b/youtube_dl/extractor/extractors.py
@ -44,6 +44,7 @@ from .anysex import AnySexIE
 from .aol import AolIE
 from .allocine import AllocineIE
 from .aliexpress import AliExpressLiveIE
 from .apa import APAIE
 from .aparat import AparatIE
 from .appleconnect import AppleConnectIE
 from .appletrailers import (
@ -53,6 +54,7 @@ from .appletrailers import (
 from .archiveorg import ArchiveOrgIE
 from .arkena import ArkenaIE
 from .ard import (
    ARDBetaMediathekIE,
    ARDIE,
    ARDMediathekIE,
 )
@ -117,6 +119,10 @@ from .bilibili import (
    BiliBiliBangumiIE,
 )
 from .biobiochiletv import BioBioChileTVIE
 from .bitchute import (
    BitChuteIE,
    BitChuteChannelIE,
 )
 from .biqle import BIQLEIE
 from .bleacherreport import (
    BleacherReportIE,
@ -137,6 +143,7 @@ from .brightcove import (
    BrightcoveLegacyIE,
    BrightcoveNewIE,
 )
 from .businessinsider import BusinessInsiderIE
 from .buzzfeed import BuzzFeedIE
 from .byutv import BYUtvIE
 from .c56 import C56IE
@ -144,6 +151,8 @@ from .camdemy import (
    CamdemyIE,
    CamdemyFolderIE
 )
 from .cammodels import CamModelsIE
 from .camtube import CamTubeIE
 from .camwithher import CamWithHerIE
 from .canalplus import CanalplusIE
 from .canalc2 import Canalc2IE
@ -195,6 +204,7 @@ from .clippit import ClippitIE
 from .cliprs import ClipRsIE
 from .clipsyndicate import ClipsyndicateIE
 from .closertotruth import CloserToTruthIE
 from .cloudflarestream import CloudflareStreamIE
 from .cloudy import CloudyIE
 from .clubic import ClubicIE
 from .clyp import ClypIE
@ -281,6 +291,7 @@ from .drtv import (
    DRTVIE,
    DRTVLiveIE,
 )
 from .dtube import DTubeIE
 from .dvtv import DVTVIE
 from .dumpert import DumpertIE
 from .defense import DefenseGouvFrIE
@ -326,10 +337,10 @@ from .espn import (
    FiveThirtyEightIE,
 )
 from .esri import EsriVideoIE
 from .etonline import ETOnlineIE
 from .europa import EuropaIE
 from .everyonesmixtape import EveryonesMixtapeIE
 from .expotv import ExpoTVIE
 from .expressen import ExpressenIE
 from .extremetube import ExtremeTubeIE
 from .eyedotv import EyedoTVIE
 from .facebook import (
@ -367,7 +378,6 @@ from .foxgay import FoxgayIE
 from .foxnews import (
    FoxNewsIE,
    FoxNewsArticleIE,
    FoxNewsInsiderIE,
 )
 from .foxsports import FoxSportsIE
 from .franceculture import FranceCultureIE
@ -377,6 +387,7 @@ from .francetv import (
    FranceTVSiteIE,
    FranceTVEmbedIE,
    FranceTVInfoIE,
    FranceTVInfoSportIE,
    FranceTVJeunesseIE,
    GenerationWhatIE,
    CultureboxIE,
@ -384,6 +395,11 @@ from .francetv import (
 from .freesound import FreesoundIE
 from .freespeech import FreespeechIE
 from .freshlive import FreshLiveIE
 from .frontendmasters import (
    FrontendMastersIE,
    FrontendMastersLessonIE,
    FrontendMastersCourseIE
 )
 from .funimation import FunimationIE
 from .funk import (
    FunkMixIE,
@ -467,10 +483,7 @@ from .imgur import (
 )
 from .ina import InaIE
 from .inc import IncIE
-from .indavideo import (
+from .indavideo import IndavideoEmbedIE
    IndavideoIE,
    IndavideoEmbedIE,
 )
 from .infoq import InfoQIE
 from .instagram import InstagramIE, InstagramUserIE
 from .internazionale import InternazionaleIE
@ -478,7 +491,10 @@ from .internetvideoarchive import InternetVideoArchiveIE
 from .iprima import IPrimaIE
 from .iqiyi import IqiyiIE
 from .ir90tv import Ir90TvIE
-from .itv import ITVIE
+from .itv import (
    ITVIE,
    ITVBTCCIE,
 )
 from .ivi import (
    IviIE,
    IviCompilationIE
@ -505,6 +521,7 @@ from .keezmovies import KeezMoviesIE
 from .ketnet import KetnetIE
 from .khanacademy import KhanAcademyIE
 from .kickstarter import KickStarterIE
 from .kinopoisk import KinoPoiskIE
 from .keek import KeekIE
 from .konserthusetplay import KonserthusetPlayIE
 from .kontrtube import KontrTubeIE
@ -577,13 +594,16 @@ from .mailru import (
    MailRuMusicIE,
    MailRuMusicSearchIE,
 )
 from .makerschannel import MakersChannelIE
 from .makertv import MakerTVIE
 from .mangomolo import (
    MangomoloVideoIE,
    MangomoloLiveIE,
 )
 from .manyvids import ManyVidsIE
 from .markiza import (
    MarkizaIE,
    MarkizaPageIE,
 )
 from .massengeschmacktv import MassengeschmackTVIE
 from .matchtv import MatchTVIE
 from .mdr import MDRIE
@ -620,7 +640,6 @@ from .mnet import MnetIE
 from .moevideo import MoeVideoIE
 from .mofosex import MofosexIE
 from .mojvideo import MojvideoIE
 from .moniker import MonikerIE
 from .morningstar import MorningstarIE
 from .motherless import (
    MotherlessIE,
@ -641,6 +660,7 @@ from .mtv import (
 from .muenchentv import MuenchenTVIE
 from .musicplayon import MusicPlayOnIE
 from .mwave import MwaveIE, MwaveMeetGreetIE
 from .mychannels import MyChannelsIE
 from .myspace import MySpaceIE, MySpaceAlbumIE
 from .myspass import MySpassIE
 from .myvi import (
@ -662,6 +682,7 @@ from .nbc import (
    NBCOlympicsIE,
    NBCOlympicsStreamIE,
    NBCSportsIE,
    NBCSportsStreamIE,
    NBCSportsVPlayerIE,
 )
 from .ndr import (
@ -701,12 +722,7 @@ from .nexx import (
 from .nfb import NFBIE
 from .nfl import NFLIE
 from .nhk import NhkVodIE
-from .nhl import (
+from .nhl import NHLIE
    NHLVideocenterIE,
    NHLNewsIE,
    NHLVideocenterCategoryIE,
    NHLIE,
 )
 from .nick import (
    NickIE,
    NickBrIE,
@ -715,10 +731,7 @@ from .nick import (
    NickRuIE,
 )
 from .niconico import NiconicoIE, NiconicoPlaylistIE
-from .ninecninemedia import (
+from .ninecninemedia import NineCNineMediaIE
    NineCNineMediaStackIE,
    NineCNineMediaIE,
 )
 from .ninegag import NineGagIE
 from .ninenow import NineNowIE
 from .nintendo import NintendoIE
@ -729,7 +742,10 @@ from .nonktube import NonkTubeIE
 from .noovo import NoovoIE
 from .normalboots import NormalbootsIE
 from .nosvideo import NosVideoIE
-from .nova import NovaIE
+from .nova import (
    NovaEmbedIE,
    NovaIE,
 )
 from .novamov import (
    AuroraVidIE,
    CloudTimeIE,
@ -761,7 +777,9 @@ from .nrk import (
    NRKSkoleIE,
    NRKTVIE,
    NRKTVDirekteIE,
    NRKTVEpisodeIE,
    NRKTVEpisodesIE,
    NRKTVSeasonIE,
    NRKTVSeriesIE,
 )
 from .ntvde import NTVDeIE
@ -806,6 +824,7 @@ from .parliamentliveuk import ParliamentLiveUKIE
 from .patreon import PatreonIE
 from .pbs import PBSIE
 from .pearvideo import PearVideoIE
 from .peertube import PeerTubeIE
 from .people import PeopleIE
 from .performgroup import PerformGroupIE
 from .periscope import (
@ -815,6 +834,10 @@ from .periscope import (
 from .philharmoniedeparis import PhilharmonieDeParisIE
 from .phoenix import PhoenixIE
 from .photobucket import PhotobucketIE
 from .picarto import (
    PicartoIE,
    PicartoVodIE,
 )
 from .piksel import PikselIE
 from .pinkbike import PinkbikeIE
 from .pladform import PladformIE
@ -846,6 +869,10 @@ from .pornhub import (
 from .pornotube import PornotubeIE
 from .pornovoisines import PornoVoisinesIE
 from .pornoxo import PornoXOIE
 from .puhutv import (
    PuhuTVIE,
    PuhuTVSerieIE,
 )
 from .presstv import PressTVIE
 from .primesharetv import PrimeShareTVIE
 from .promptfile import PromptFileIE
@ -877,7 +904,10 @@ from .rai import (
    RaiPlayPlaylistIE,
    RaiIE,
 )
-from .raywenderlich import RayWenderlichIE
+from .raywenderlich import (
    RayWenderlichIE,
    RayWenderlichCourseIE,
 )
 from .rbmaradio import RBMARadioIE
 from .rds import RDSIE
 from .redbulltv import RedBullTVIE
@ -1007,7 +1037,10 @@ from .spankbang import SpankBangIE
 from .spankwire import SpankwireIE
 from .spiegel import SpiegelIE, SpiegelArticleIE
 from .spiegeltv import SpiegeltvIE
-from .spike import SpikeIE
+from .spike import (
    BellatorIE,
    ParamountNetworkIE,
 )
 from .stitcher import StitcherIE
 from .sport5 import Sport5IE
 from .sportbox import SportBoxEmbedIE
@ -1030,7 +1063,9 @@ from .stretchinternet import StretchInternetIE
 from .sunporno import SunPornoIE
 from .svt import (
    SVTIE,
    SVTPageIE,
    SVTPlayIE,
    SVTSeriesIE,
 )
 from .swrmediathek import SWRMediathekIE
 from .syfy import SyfyIE
@ -1051,6 +1086,7 @@ from .teachingchannel import TeachingChannelIE
 from .teamcoco import TeamcocoIE
 from .techtalks import TechTalksIE
 from .ted import TEDIE
 from .tele5 import Tele5IE
 from .tele13 import Tele13IE
 from .telebruxelles import TeleBruxellesIE
 from .telecinco import TelecincoIE
@ -1133,10 +1169,12 @@ from .tvc import (
 from .tvigle import TvigleIE
 from .tvland import TVLandIE
 from .tvn24 import TVN24IE
 from .tvnet import TVNetIE
 from .tvnoe import TVNoeIE
 from .tvnow import (
    TVNowIE,
    TVNowListIE,
    TVNowShowIE,
 )
 from .tvp import (
    TVPEmbedIE,
@ -1146,6 +1184,7 @@ from .tvp import (
 from .tvplay import (
    TVPlayIE,
    ViafreeIE,
    TVPlayHomeIE,
 )
 from .tvplayer import TVPlayerIE
 from .tweakers import TweakersIE
@ -1267,6 +1306,7 @@ from .viki import (
    VikiIE,
    VikiChannelIE,
 )
 from .viqeo import ViqeoIE
 from .viu import (
    ViuIE,
    ViuPlaylistIE,
@ -1392,6 +1432,7 @@ from .younow import (
    YouNowMomentIE,
 )
 from .youporn import YouPornIE
 from .yourporn import YourPornIE
 from .yourupload import YourUploadIE
 from .youtube import (
    YoutubeIE,
@ -1414,5 +1455,23 @@ from .youtube import (
 )
 from .zapiks import ZapiksIE
 from .zaq1 import Zaq1IE
 from .zattoo import (
    BBVTVIE,
    EinsUndEinsTVIE,
    EWETVIE,
    GlattvisionTVIE,
    MNetTVIE,
    MyVisionTVIE,
    NetPlusIE,
    OsnatelTVIE,
    QuantumTVIE,
    QuicklineIE,
    QuicklineLiveIE,
    SAKTVIE,
    VTXTVIE,
    WalyTVIE,
    ZattooIE,
    ZattooLiveIE,
 )
 from .zdf import ZDFIE, ZDFChannelIE
 from .zingmp3 import ZingMp3IE
--- a/youtube_dl/extractor/extremetube.py
+++ b/youtube_dl/extractor/extremetube.py
@ -8,12 +8,12 @@ class ExtremeTubeIE(KeezMoviesIE):
    _VALID_URL = r'https?://(?:www\.)?extremetube\.com/(?:[^/]+/)?video/(?P<id>[^/#?&]+)'
    _TESTS = [{
        'url': 'http://www.extremetube.com/video/music-video-14-british-euro-brit-european-cumshots-swallow-652431',
-        'md5': '1fb9228f5e3332ec8c057d6ac36f33e0',
+        'md5': '92feaafa4b58e82f261e5419f39c60cb',
        'info_dict': {
            'id': 'music-video-14-british-euro-brit-european-cumshots-swallow-652431',
            'ext': 'mp4',
            'title': 'Music Video 14 british euro brit european cumshots swallow',
-            'uploader': 'unknown',
+            'uploader': 'anonim',
            'view_count': int,
            'age_limit': 18,
        }
@ -36,10 +36,10 @@ class ExtremeTubeIE(KeezMoviesIE):
                r'<h1[^>]+title="([^"]+)"[^>]*>', webpage, 'title')
        uploader = self._html_search_regex(
-            r'Uploaded by:\s*</strong>\s*(.+?)\s*</div>',
+            r'Uploaded by:\s*</[^>]+>\s*<a[^>]+>(.+?)</a>',
            webpage, 'uploader', fatal=False)
        view_count = str_to_int(self._search_regex(
-            r'Views:\s*</strong>\s*<span>([\d,\.]+)</span>',
+            r'Views:\s*</[^>]+>\s*<[^>]+>([\d,\.]+)</',
            webpage, 'view count', fatal=False))
        info.update({
--- a/youtube_dl/extractor/facebook.py
+++ b/youtube_dl/extractor/facebook.py
@ -20,6 +20,7 @@ from ..utils import (
    int_or_none,
    js_to_json,
    limit_length,
    parse_count,
    sanitized_Request,
    try_get,
    urlencode_postdata,
@ -56,6 +57,7 @@ class FacebookIE(InfoExtractor):
    _CHROME_USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.97 Safari/537.36'
    _VIDEO_PAGE_TEMPLATE = 'https://www.facebook.com/video/video.php?v=%s'
    _VIDEO_PAGE_TAHOE_TEMPLATE = 'https://www.facebook.com/video/tahoe/async/%s/?chain=true&isvideo=true'
    _TESTS = [{
        'url': 'https://www.facebook.com/video.php?v=637842556329505&fref=nf',
@ -74,7 +76,7 @@ class FacebookIE(InfoExtractor):
        'info_dict': {
            'id': '274175099429670',
            'ext': 'mp4',
-            'title': 'Asif Nawab Butt posted a video to his Timeline.',
+            'title': 're:^Asif Nawab Butt posted a video',
            'uploader': 'Asif Nawab Butt',
            'upload_date': '20140506',
            'timestamp': 1399398998,
@ -132,7 +134,7 @@ class FacebookIE(InfoExtractor):
    }, {
        # have 1080P, but only up to 720p in swf params
        'url': 'https://www.facebook.com/cnn/videos/10155529876156509/',
-        'md5': '0d9813160b146b3bc8744e006027fcc6',
+        'md5': '9571fae53d4165bbbadb17a94651dcdc',
        'info_dict': {
            'id': '10155529876156509',
            'ext': 'mp4',
@ -141,6 +143,7 @@ class FacebookIE(InfoExtractor):
            'upload_date': '20161030',
            'uploader': 'CNN',
            'thumbnail': r're:^https?://.*',
            'view_count': int,
        },
    }, {
        # bigPipe.onPageletArrive ... onPageletArrive pagelet_group_mall
@ -148,7 +151,7 @@ class FacebookIE(InfoExtractor):
        'info_dict': {
            'id': '1417995061575415',
            'ext': 'mp4',
-            'title': 'md5:a7b86ca673f51800cd54687b7f4012fe',
+            'title': 'md5:1db063d6a8c13faa8da727817339c857',
            'timestamp': 1486648217,
            'upload_date': '20170209',
            'uploader': 'Yaroslav Korpan',
@ -175,7 +178,7 @@ class FacebookIE(InfoExtractor):
        'info_dict': {
            'id': '1396382447100162',
            'ext': 'mp4',
-            'title': 'md5:e2d2700afdf84e121f5d0f999bad13a3',
+            'title': 'md5:19a428bbde91364e3de815383b54a235',
            'timestamp': 1486035494,
            'upload_date': '20170202',
            'uploader': 'Elisabeth Ahtn',
@ -208,6 +211,17 @@ class FacebookIE(InfoExtractor):
        # no title
        'url': 'https://www.facebook.com/onlycleverentertainment/videos/1947995502095005/',
        'only_matching': True,
    }, {
        'url': 'https://www.facebook.com/WatchESLOne/videos/359649331226507/',
        'info_dict': {
            'id': '359649331226507',
            'ext': 'mp4',
            'title': '#ESLOne VoD - Birmingham Finals Day#1 Fnatic vs. @Evil Geniuses',
            'uploader': 'ESL One Dota 2',
        },
        'params': {
            'skip_download': True,
        },
    }]
    @staticmethod
@ -226,7 +240,7 @@ class FacebookIE(InfoExtractor):
        return urls
    def _login(self):
-        (useremail, password) = self._get_login_info()
+        useremail, password = self._get_login_info()
        if useremail is None:
            return
@ -312,16 +326,18 @@ class FacebookIE(InfoExtractor):
        if server_js_data:
            video_data = extract_video_data(server_js_data.get('instances', []))
        def extract_from_jsmods_instances(js_data):
            if js_data:
                return extract_video_data(try_get(
                    js_data, lambda x: x['jsmods']['instances'], list) or [])
        if not video_data:
            server_js_data = self._parse_json(
                self._search_regex(
                    r'bigPipe\.onPageletArrive\(({.+?})\)\s*;\s*}\s*\)\s*,\s*["\']onPageletArrive\s+(?:stream_pagelet|pagelet_group_mall|permalink_video_pagelet)',
                    webpage, 'js data', default='{}'),
                video_id, transform_source=js_to_json, fatal=False)
-            if server_js_data:
+            video_data = extract_from_jsmods_instances(server_js_data)
                video_data = extract_video_data(try_get(
                    server_js_data, lambda x: x['jsmods']['instances'],
                    list) or [])
        if not video_data:
            if not fatal_if_no_video:
@ -333,8 +349,35 @@ class FacebookIE(InfoExtractor):
                    expected=True)
            elif '>You must log in to continue' in webpage:
                self.raise_login_required()
-            else:
+
-                raise ExtractorError('Cannot parse data')
+            # Video info not in first request, do a secondary request using
            # tahoe player specific URL
            tahoe_data = self._download_webpage(
                self._VIDEO_PAGE_TAHOE_TEMPLATE % video_id, video_id,
                data=urlencode_postdata({
                    '__a': 1,
                    '__pc': self._search_regex(
                        r'pkg_cohort["\']\s*:\s*["\'](.+?)["\']', webpage,
                        'pkg cohort', default='PHASED:DEFAULT'),
                    '__rev': self._search_regex(
                        r'client_revision["\']\s*:\s*(\d+),', webpage,
                        'client revision', default='3944515'),
                    'fb_dtsg': self._search_regex(
                        r'"DTSGInitialData"\s*,\s*\[\]\s*,\s*{\s*"token"\s*:\s*"([^"]+)"',
                        webpage, 'dtsg token', default=''),
                }),
                headers={
                    'Content-Type': 'application/x-www-form-urlencoded',
                })
            tahoe_js_data = self._parse_json(
                self._search_regex(
                    r'for\s+\(\s*;\s*;\s*\)\s*;(.+)', tahoe_data,
                    'tahoe js data', default='{}'),
                video_id, fatal=False)
            video_data = extract_from_jsmods_instances(tahoe_js_data)
        if not video_data:
            raise ExtractorError('Cannot parse data')
        formats = []
        for f in video_data:
@ -380,12 +423,17 @@ class FacebookIE(InfoExtractor):
            video_title = 'Facebook video #%s' % video_id
        uploader = clean_html(get_element_by_id(
            'fbPhotoPageAuthorName', webpage)) or self._search_regex(
-            r'ownerName\s*:\s*"([^"]+)"', webpage, 'uploader', fatal=False)
+            r'ownerName\s*:\s*"([^"]+)"', webpage, 'uploader',
            fatal=False) or self._og_search_title(webpage, fatal=False)
        timestamp = int_or_none(self._search_regex(
            r'<abbr[^>]+data-utime=["\'](\d+)', webpage,
            'timestamp', default=None))
        thumbnail = self._og_search_thumbnail(webpage)
        view_count = parse_count(self._search_regex(
            r'\bviewCount\s*:\s*["\']([\d,.]+)', webpage, 'view count',
            default=None))
        info_dict = {
            'id': video_id,
            'title': video_title,
@ -393,6 +441,7 @@ class FacebookIE(InfoExtractor):
            'uploader': uploader,
            'timestamp': timestamp,
            'thumbnail': thumbnail,
            'view_count': view_count,
        }
        return webpage, info_dict
--- a/youtube_dl/extractor/fc2.py
+++ b/youtube_dl/extractor/fc2.py
@ -46,7 +46,7 @@ class FC2IE(InfoExtractor):
    }]
    def _login(self):
-        (username, password) = self._get_login_info()
+        username, password = self._get_login_info()
        if username is None or password is None:
            return False
--- a/youtube_dl/extractor/firsttv.py
+++ b/youtube_dl/extractor/firsttv.py
@ -10,6 +10,7 @@ from ..utils import (
    int_or_none,
    qualities,
    unified_strdate,
    url_or_none,
 )
@ -88,8 +89,8 @@ class FirstTVIE(InfoExtractor):
            formats = []
            path = None
            for f in item.get('mbr', []):
-                src = f.get('src')
+                src = url_or_none(f.get('src'))
-                if not src or not isinstance(src, compat_str):
+                if not src:
                    continue
                tbr = int_or_none(self._search_regex(
                    r'_(\d{3,})\.mp4', src, 'tbr', default=None))
--- a/youtube_dl/extractor/fourtube.py
+++ b/youtube_dl/extractor/fourtube.py
@ -3,15 +3,45 @@ from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
-from ..compat import compat_urlparse
+from ..compat import (
    compat_b64decode,
    compat_str,
    compat_urllib_parse_unquote,
    compat_urlparse,
 )
 from ..utils import (
    int_or_none,
    parse_duration,
    parse_iso8601,
    str_or_none,
    str_to_int,
    try_get,
    unified_timestamp,
    url_or_none,
 )
 class FourTubeBaseIE(InfoExtractor):
    _TKN_HOST = 'tkn.kodicdn.com'
    def _extract_formats(self, url, video_id, media_id, sources):
        token_url = 'https://%s/%s/desktop/%s' % (
            self._TKN_HOST, media_id, '+'.join(sources))
        parsed_url = compat_urlparse.urlparse(url)
        tokens = self._download_json(token_url, video_id, data=b'', headers={
            'Origin': '%s://%s' % (parsed_url.scheme, parsed_url.hostname),
            'Referer': url,
        })
        formats = [{
            'url': tokens[format]['token'],
            'format_id': format + 'p',
            'resolution': format + 'p',
            'quality': int(format),
        } for format in sources]
        self._sort_formats(formats)
        return formats
    def _real_extract(self, url):
        mobj = re.match(self._VALID_URL, url)
        kind, video_id, display_id = mobj.group('kind', 'id', 'display_id')
@ -68,21 +98,7 @@ class FourTubeBaseIE(InfoExtractor):
            media_id = params[0]
            sources = ['%s' % p for p in params[2]]
-        token_url = 'https://tkn.kodicdn.com/{0}/desktop/{1}'.format(
+        formats = self._extract_formats(url, video_id, media_id, sources)
            media_id, '+'.join(sources))
        parsed_url = compat_urlparse.urlparse(url)
        tokens = self._download_json(token_url, video_id, data=b'', headers={
            'Origin': '%s://%s' % (parsed_url.scheme, parsed_url.hostname),
            'Referer': url,
        })
        formats = [{
            'url': tokens[format]['token'],
            'format_id': format + 'p',
            'resolution': format + 'p',
            'quality': int(format),
        } for format in sources]
        self._sort_formats(formats)
        return {
            'id': video_id,
@ -164,6 +180,7 @@ class FuxIE(FourTubeBaseIE):
 class PornTubeIE(FourTubeBaseIE):
    _VALID_URL = r'https?://(?:(?P<kind>www|m)\.)?porntube\.com/(?:videos/(?P<display_id>[^/]+)_|embed/)(?P<id>\d+)'
    _URL_TEMPLATE = 'https://www.porntube.com/videos/video_%s'
    _TKN_HOST = 'tkn.porntube.com'
    _TESTS = [{
        'url': 'https://www.porntube.com/videos/teen-couple-doing-anal_7089759',
        'info_dict': {
@ -171,13 +188,32 @@ class PornTubeIE(FourTubeBaseIE):
            'ext': 'mp4',
            'title': 'Teen couple doing anal',
            'uploader': 'Alexy',
-            'uploader_id': 'Alexy',
+            'uploader_id': '91488',
            'upload_date': '20150606',
            'timestamp': 1433595647,
            'duration': 5052,
            'view_count': int,
            'like_count': int,
-            'categories': list,
+            'age_limit': 18,
        },
        'params': {
            'skip_download': True,
        },
    }, {
        'url': 'https://www.porntube.com/videos/squirting-teen-ballerina-ecg_1331406',
        'info_dict': {
            'id': '1331406',
            'ext': 'mp4',
            'title': 'Squirting Teen Ballerina on ECG',
            'uploader': 'Exploited College Girls',
            'uploader_id': '665',
            'channel': 'Exploited College Girls',
            'channel_id': '665',
            'upload_date': '20130920',
            'timestamp': 1379685485,
            'duration': 851,
            'view_count': int,
            'like_count': int,
            'age_limit': 18,
        },
        'params': {
@ -191,6 +227,55 @@ class PornTubeIE(FourTubeBaseIE):
        'only_matching': True,
    }]
    def _real_extract(self, url):
        mobj = re.match(self._VALID_URL, url)
        video_id, display_id = mobj.group('id', 'display_id')
        webpage = self._download_webpage(url, display_id)
        video = self._parse_json(
            self._search_regex(
                r'INITIALSTATE\s*=\s*(["\'])(?P<value>(?:(?!\1).)+)\1',
                webpage, 'data', group='value'), video_id,
            transform_source=lambda x: compat_urllib_parse_unquote(
                compat_b64decode(x).decode('utf-8')))['page']['video']
        title = video['title']
        media_id = video['mediaId']
        sources = [compat_str(e['height'])
                   for e in video['encodings'] if e.get('height')]
        formats = self._extract_formats(url, video_id, media_id, sources)
        thumbnail = url_or_none(video.get('masterThumb'))
        uploader = try_get(video, lambda x: x['user']['username'], compat_str)
        uploader_id = str_or_none(try_get(
            video, lambda x: x['user']['id'], int))
        channel = try_get(video, lambda x: x['channel']['name'], compat_str)
        channel_id = str_or_none(try_get(
            video, lambda x: x['channel']['id'], int))
        like_count = int_or_none(video.get('likes'))
        dislike_count = int_or_none(video.get('dislikes'))
        view_count = int_or_none(video.get('playsQty'))
        duration = int_or_none(video.get('durationInSeconds'))
        timestamp = unified_timestamp(video.get('publishedAt'))
        return {
            'id': video_id,
            'title': title,
            'formats': formats,
            'thumbnail': thumbnail,
            'uploader': uploader or channel,
            'uploader_id': uploader_id or channel_id,
            'channel': channel,
            'channel_id': channel_id,
            'timestamp': timestamp,
            'like_count': like_count,
            'dislike_count': dislike_count,
            'view_count': view_count,
            'duration': duration,
            'age_limit': 18,
        }
 class PornerBrosIE(FourTubeBaseIE):
    _VALID_URL = r'https?://(?:(?P<kind>www|m)\.)?pornerbros\.com/(?:videos/(?P<display_id>[^/]+)_|embed/)(?P<id>\d+)'
--- a/youtube_dl/extractor/foxnews.py
+++ b/youtube_dl/extractor/foxnews.py
@ -58,6 +58,14 @@ class FoxNewsIE(AMPIE):
        },
    ]
    @staticmethod
    def _extract_urls(webpage):
        return [
            mobj.group('url')
            for mobj in re.finditer(
                r'<(?:amp-)?iframe[^>]+\bsrc=(["\'])(?P<url>(?:https?:)?//video\.foxnews\.com/v/video-embed\.html?.*?\bvideo_id=\d+.*?)\1',
                webpage)]
    def _real_extract(self, url):
        host, video_id = re.match(self._VALID_URL, url).groups()
@ -68,21 +76,41 @@ class FoxNewsIE(AMPIE):
 class FoxNewsArticleIE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?foxnews\.com/(?!v)([^/]+/)+(?P<id>[a-z-]+)'
+    _VALID_URL = r'https?://(?:www\.)?(?:insider\.)?foxnews\.com/(?!v)([^/]+/)+(?P<id>[a-z-]+)'
    IE_NAME = 'foxnews:article'
-    _TEST = {
+    _TESTS = [{
        # data-video-id
        'url': 'http://www.foxnews.com/politics/2016/09/08/buzz-about-bud-clinton-camp-denies-claims-wore-earpiece-at-forum.html',
-        'md5': '62aa5a781b308fdee212ebb6f33ae7ef',
+        'md5': '83d44e1aff1433e7a29a7b537d1700b5',
        'info_dict': {
            'id': '5116295019001',
            'ext': 'mp4',
            'title': 'Trump and Clinton asked to defend positions on Iraq War',
            'description': 'Veterans react on \'The Kelly File\'',
-            'timestamp': 1473299755,
+            'timestamp': 1473301045,
            'upload_date': '20160908',
        },
-    }
+    }, {
        # iframe embed
        'url': 'http://www.foxnews.com/us/2018/03/09/parkland-survivor-kyle-kashuv-on-meeting-trump-his-app-to-prevent-another-school-shooting.amp.html?__twitter_impression=true',
        'info_dict': {
            'id': '5748266721001',
            'ext': 'flv',
            'title': 'Kyle Kashuv has a positive message for the Trump White House',
            'description': 'Marjory Stoneman Douglas student disagrees with classmates.',
            'thumbnail': r're:^https?://.*\.jpg$',
            'duration': 229,
            'timestamp': 1520594670,
            'upload_date': '20180309',
        },
        'params': {
            'skip_download': True,
        },
    }, {
        'url': 'http://insider.foxnews.com/2016/08/25/univ-wisconsin-student-group-pushing-silence-certain-words',
        'only_matching': True,
    }]
    def _real_extract(self, url):
        display_id = self._match_id(url)
@ -90,51 +118,10 @@ class FoxNewsArticleIE(InfoExtractor):
        video_id = self._html_search_regex(
            r'data-video-id=([\'"])(?P<id>[^\'"]+)\1',
-            webpage, 'video ID', group='id')
+            webpage, 'video ID', group='id', default=None)
        if video_id:
            return self.url_result(
                'http://video.foxnews.com/v/' + video_id, FoxNewsIE.ie_key())
        return self.url_result(
-            'http://video.foxnews.com/v/' + video_id,
+            FoxNewsIE._extract_urls(webpage)[0], FoxNewsIE.ie_key())
            FoxNewsIE.ie_key())
 class FoxNewsInsiderIE(InfoExtractor):
    _VALID_URL = r'https?://insider\.foxnews\.com/([^/]+/)+(?P<id>[a-z-]+)'
    IE_NAME = 'foxnews:insider'
    _TEST = {
        'url': 'http://insider.foxnews.com/2016/08/25/univ-wisconsin-student-group-pushing-silence-certain-words',
        'md5': 'a10c755e582d28120c62749b4feb4c0c',
        'info_dict': {
            'id': '5099377331001',
            'display_id': 'univ-wisconsin-student-group-pushing-silence-certain-words',
            'ext': 'mp4',
            'title': 'Student Group: Saying \'Politically Correct,\' \'Trash\' and \'Lame\' Is Offensive',
            'description': 'Is campus censorship getting out of control?',
            'timestamp': 1472168725,
            'upload_date': '20160825',
            'thumbnail': r're:^https?://.*\.jpg$',
        },
        'params': {
            # m3u8 download
            'skip_download': True,
        },
        'add_ie': [FoxNewsIE.ie_key()],
    }
    def _real_extract(self, url):
        display_id = self._match_id(url)
        webpage = self._download_webpage(url, display_id)
        embed_url = self._html_search_meta('embedUrl', webpage, 'embed URL')
        title = self._og_search_title(webpage)
        description = self._og_search_description(webpage)
        return {
            '_type': 'url_transparent',
            'ie_key': FoxNewsIE.ie_key(),
            'url': embed_url,
            'display_id': display_id,
            'title': title,
            'description': description,
        }
--- a/youtube_dl/extractor/francetv.py
+++ b/youtube_dl/extractor/francetv.py
@ -16,6 +16,7 @@ from ..utils import (
    int_or_none,
    parse_duration,
    try_get,
    url_or_none,
 )
 from .dailymotion import DailymotionIE
@ -115,14 +116,13 @@ class FranceTVIE(InfoExtractor):
        def sign(manifest_url, manifest_id):
            for host in ('hdfauthftv-a.akamaihd.net', 'hdfauth.francetv.fr'):
-                signed_url = self._download_webpage(
+                signed_url = url_or_none(self._download_webpage(
                    'https://%s/esi/TA' % host, video_id,
                    'Downloading signed %s manifest URL' % manifest_id,
                    fatal=False, query={
                        'url': manifest_url,
-                    })
+                    }))
-                if (signed_url and isinstance(signed_url, compat_str) and
+                if signed_url:
                        re.search(r'^(?:https?:)?//', signed_url)):
                    return signed_url
            return manifest_url
@ -379,6 +379,31 @@ class FranceTVInfoIE(FranceTVBaseInfoExtractor):
        return self._make_url_result(video_id, catalogue)
 class FranceTVInfoSportIE(FranceTVBaseInfoExtractor):
    IE_NAME = 'sport.francetvinfo.fr'
    _VALID_URL = r'https?://sport\.francetvinfo\.fr/(?:[^/]+/)*(?P<id>[^/?#&]+)'
    _TESTS = [{
        'url': 'https://sport.francetvinfo.fr/les-jeux-olympiques/retour-sur-les-meilleurs-moments-de-pyeongchang-2018',
        'info_dict': {
            'id': '6e49080e-3f45-11e8-b459-000d3a2439ea',
            'ext': 'mp4',
            'title': 'Retour sur les meilleurs moments de Pyeongchang 2018',
            'timestamp': 1523639962,
            'upload_date': '20180413',
        },
        'params': {
            'skip_download': True,
        },
        'add_ie': [FranceTVIE.ie_key()],
    }]
    def _real_extract(self, url):
        display_id = self._match_id(url)
        webpage = self._download_webpage(url, display_id)
        video_id = self._search_regex(r'data-video="([^"]+)"', webpage, 'video_id')
        return self._make_url_result(video_id, 'Sport-web')
 class GenerationWhatIE(InfoExtractor):
    IE_NAME = 'france2.fr:generation-what'
    _VALID_URL = r'https?://generation-what\.francetv\.fr/[^/]+/video/(?P<id>[^/?#&]+)'
--- a/youtube_dl/extractor/frontendmasters.py
+++ b/youtube_dl/extractor/frontendmasters.py
@ -0,0 +1,263 @@
 # coding: utf-8
 from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
 from ..compat import (
    compat_str,
    compat_urlparse,
 )
 from ..utils import (
    ExtractorError,
    parse_duration,
    url_or_none,
    urlencode_postdata,
 )
 class FrontendMastersBaseIE(InfoExtractor):
    _API_BASE = 'https://api.frontendmasters.com/v1/kabuki'
    _LOGIN_URL = 'https://frontendmasters.com/login/'
    _NETRC_MACHINE = 'frontendmasters'
    _QUALITIES = {
        'low': {'width': 480, 'height': 360},
        'mid': {'width': 1280, 'height': 720},
        'high': {'width': 1920, 'height': 1080}
    }
    def _real_initialize(self):
        self._login()
    def _login(self):
        (username, password) = self._get_login_info()
        if username is None:
            return
        login_page = self._download_webpage(
            self._LOGIN_URL, None, 'Downloading login page')
        login_form = self._hidden_inputs(login_page)
        login_form.update({
            'username': username,
            'password': password
        })
        post_url = self._search_regex(
            r'<form[^>]+action=(["\'])(?P<url>.+?)\1', login_page,
            'post_url', default=self._LOGIN_URL, group='url')
        if not post_url.startswith('http'):
            post_url = compat_urlparse.urljoin(self._LOGIN_URL, post_url)
        response = self._download_webpage(
            post_url, None, 'Logging in', data=urlencode_postdata(login_form),
            headers={'Content-Type': 'application/x-www-form-urlencoded'})
        # Successful login
        if any(p in response for p in (
                'wp-login.php?action=logout', '>Logout')):
            return
        error = self._html_search_regex(
            r'class=(["\'])(?:(?!\1).)*\bMessageAlert\b(?:(?!\1).)*\1[^>]*>(?P<error>[^<]+)<',
            response, 'error message', default=None, group='error')
        if error:
            raise ExtractorError('Unable to login: %s' % error, expected=True)
        raise ExtractorError('Unable to log in')
 class FrontendMastersPageBaseIE(FrontendMastersBaseIE):
    def _download_course(self, course_name, url):
        return self._download_json(
            '%s/courses/%s' % (self._API_BASE, course_name), course_name,
            'Downloading course JSON', headers={'Referer': url})
    @staticmethod
    def _extract_chapters(course):
        chapters = []
        lesson_elements = course.get('lessonElements')
        if isinstance(lesson_elements, list):
            chapters = [url_or_none(e) for e in lesson_elements if url_or_none(e)]
        return chapters
    @staticmethod
    def _extract_lesson(chapters, lesson_id, lesson):
        title = lesson.get('title') or lesson_id
        display_id = lesson.get('slug')
        description = lesson.get('description')
        thumbnail = lesson.get('thumbnail')
        chapter_number = None
        index = lesson.get('index')
        element_index = lesson.get('elementIndex')
        if (isinstance(index, int) and isinstance(element_index, int) and
                index < element_index):
            chapter_number = element_index - index
        chapter = (chapters[chapter_number - 1]
                   if chapter_number - 1 < len(chapters) else None)
        duration = None
        timestamp = lesson.get('timestamp')
        if isinstance(timestamp, compat_str):
            mobj = re.search(
                r'(?P<start>\d{1,2}:\d{1,2}:\d{1,2})\s*-(?P<end>\s*\d{1,2}:\d{1,2}:\d{1,2})',
                timestamp)
            if mobj:
                duration = parse_duration(mobj.group('end')) - parse_duration(
                    mobj.group('start'))
        return {
            '_type': 'url_transparent',
            'url': 'frontendmasters:%s' % lesson_id,
            'ie_key': FrontendMastersIE.ie_key(),
            'id': lesson_id,
            'display_id': display_id,
            'title': title,
            'description': description,
            'thumbnail': thumbnail,
            'duration': duration,
            'chapter': chapter,
            'chapter_number': chapter_number,
        }
 class FrontendMastersIE(FrontendMastersBaseIE):
    _VALID_URL = r'(?:frontendmasters:|https?://api\.frontendmasters\.com/v\d+/kabuki/video/)(?P<id>[^/]+)'
    _TESTS = [{
        'url': 'https://api.frontendmasters.com/v1/kabuki/video/a2qogef6ba',
        'md5': '7f161159710d6b7016a4f4af6fcb05e2',
        'info_dict': {
            'id': 'a2qogef6ba',
            'ext': 'mp4',
            'title': 'a2qogef6ba',
        },
        'skip': 'Requires FrontendMasters account credentials',
    }, {
        'url': 'frontendmasters:a2qogef6ba',
        'only_matching': True,
    }]
    def _real_extract(self, url):
        lesson_id = self._match_id(url)
        source_url = '%s/video/%s/source' % (self._API_BASE, lesson_id)
        formats = []
        for ext in ('webm', 'mp4'):
            for quality in ('low', 'mid', 'high'):
                resolution = self._QUALITIES[quality].copy()
                format_id = '%s-%s' % (ext, quality)
                format_url = self._download_json(
                    source_url, lesson_id,
                    'Downloading %s source JSON' % format_id, query={
                        'f': ext,
                        'r': resolution['height'],
                    }, headers={
                        'Referer': url,
                    }, fatal=False)['url']
                if not format_url:
                    continue
                f = resolution.copy()
                f.update({
                    'url': format_url,
                    'ext': ext,
                    'format_id': format_id,
                })
                formats.append(f)
        self._sort_formats(formats)
        subtitles = {
            'en': [{
                'url': '%s/transcripts/%s.vtt' % (self._API_BASE, lesson_id),
            }]
        }
        return {
            'id': lesson_id,
            'title': lesson_id,
            'formats': formats,
            'subtitles': subtitles
        }
 class FrontendMastersLessonIE(FrontendMastersPageBaseIE):
    _VALID_URL = r'https?://(?:www\.)?frontendmasters\.com/courses/(?P<course_name>[^/]+)/(?P<lesson_name>[^/]+)'
    _TEST = {
        'url': 'https://frontendmasters.com/courses/web-development/tools',
        'info_dict': {
            'id': 'a2qogef6ba',
            'display_id': 'tools',
            'ext': 'mp4',
            'title': 'Tools',
            'description': 'md5:82c1ea6472e88ed5acd1829fe992e4f7',
            'thumbnail': r're:^https?://.*\.jpg$',
            'chapter': 'Introduction',
            'chapter_number': 1,
        },
        'params': {
            'skip_download': True,
        },
        'skip': 'Requires FrontendMasters account credentials',
    }
    def _real_extract(self, url):
        mobj = re.match(self._VALID_URL, url)
        course_name, lesson_name = mobj.group('course_name', 'lesson_name')
        course = self._download_course(course_name, url)
        lesson_id, lesson = next(
            (video_id, data)
            for video_id, data in course['lessonData'].items()
            if data.get('slug') == lesson_name)
        chapters = self._extract_chapters(course)
        return self._extract_lesson(chapters, lesson_id, lesson)
 class FrontendMastersCourseIE(FrontendMastersPageBaseIE):
    _VALID_URL = r'https?://(?:www\.)?frontendmasters\.com/courses/(?P<id>[^/]+)'
    _TEST = {
        'url': 'https://frontendmasters.com/courses/web-development/',
        'info_dict': {
            'id': 'web-development',
            'title': 'Introduction to Web Development',
            'description': 'md5:9317e6e842098bf725d62360e52d49a6',
        },
        'playlist_count': 81,
        'skip': 'Requires FrontendMasters account credentials',
    }
    @classmethod
    def suitable(cls, url):
        return False if FrontendMastersLessonIE.suitable(url) else super(
            FrontendMastersBaseIE, cls).suitable(url)
    def _real_extract(self, url):
        course_name = self._match_id(url)
        course = self._download_course(course_name, url)
        chapters = self._extract_chapters(course)
        lessons = sorted(
            course['lessonData'].values(), key=lambda data: data['index'])
        entries = []
        for lesson in lessons:
            lesson_name = lesson.get('slug')
            if not lesson_name:
                continue
            lesson_id = lesson.get('hash') or lesson.get('statsId')
            entries.append(self._extract_lesson(chapters, lesson_id, lesson))
        title = course.get('title')
        description = course.get('description')
        return self.playlist_result(entries, course_name, title, description)
--- a/youtube_dl/extractor/funimation.py
+++ b/youtube_dl/extractor/funimation.py
@ -51,7 +51,7 @@ class FunimationIE(InfoExtractor):
    }]
    def _login(self):
-        (username, password) = self._get_login_info()
+        username, password = self._get_login_info()
        if username is None:
            return
        try:
--- a/youtube_dl/extractor/funk.py
+++ b/youtube_dl/extractor/funk.py
@ -1,14 +1,32 @@
 # coding: utf-8
 from __future__ import unicode_literals
 import itertools
 import re
 from .common import InfoExtractor
 from .nexx import NexxIE
-from ..utils import int_or_none
+from ..compat import compat_str
 from ..utils import (
    int_or_none,
    try_get,
 )
 class FunkBaseIE(InfoExtractor):
    _HEADERS = {
        'Accept': '*/*',
        'Accept-Language': 'en-US,en;q=0.9,ru;q=0.8',
        'authorization': 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnROYW1lIjoid2ViYXBwLXYzMSIsInNjb3BlIjoic3RhdGljLWNvbnRlbnQtYXBpLGN1cmF0aW9uLWFwaSxuZXh4LWNvbnRlbnQtYXBpLXYzMSx3ZWJhcHAtYXBpIn0.mbuG9wS9Yf5q6PqgR4fiaRFIagiHk9JhwoKES7ksVX4',
    }
    _AUTH = 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnROYW1lIjoid2ViYXBwLXYzMSIsInNjb3BlIjoic3RhdGljLWNvbnRlbnQtYXBpLGN1cmF0aW9uLWFwaSxuZXh4LWNvbnRlbnQtYXBpLXYzMSx3ZWJhcHAtYXBpIn0.mbuG9wS9Yf5q6PqgR4fiaRFIagiHk9JhwoKES7ksVX4'
    @staticmethod
    def _make_headers(referer):
        headers = FunkBaseIE._HEADERS.copy()
        headers['Referer'] = referer
        return headers
    def _make_url_result(self, video):
        return {
            '_type': 'url_transparent',
@ -45,19 +63,19 @@ class FunkMixIE(FunkBaseIE):
        lists = self._download_json(
            'https://www.funk.net/api/v3.1/curation/curatedLists/',
-            mix_id, headers={
+            mix_id, headers=self._make_headers(url), query={
                'authorization': 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnROYW1lIjoiY3VyYXRpb24tdG9vbC12Mi4wIiwic2NvcGUiOiJzdGF0aWMtY29udGVudC1hcGksY3VyYXRpb24tc2VydmljZSxzZWFyY2gtYXBpIn0.SGCC1IXHLtZYoo8PvRKlU2gXH1su8YSu47sB3S4iXBI',
                'Referer': url,
            }, query={
                'size': 100,
-            })['result']['lists']
+            })['_embedded']['curatedListList']
        metas = next(
            l for l in lists
            if mix_id in (l.get('entityId'), l.get('alias')))['videoMetas']
        video = next(
            meta['videoDataDelegate']
-            for meta in metas if meta.get('alias') == alias)
+            for meta in metas
            if try_get(
                meta, lambda x: x['videoDataDelegate']['alias'],
                compat_str) == alias)
        return self._make_url_result(video)
@ -77,6 +95,20 @@ class FunkChannelIE(FunkBaseIE):
        'params': {
            'skip_download': True,
        },
    }, {
        # only available via byIdList API
        'url': 'https://www.funk.net/channel/informr/martin-sonneborn-erklaert-die-eu',
        'info_dict': {
            'id': '205067',
            'ext': 'mp4',
            'title': 'Martin Sonneborn erklärt die EU',
            'description': 'md5:050f74626e4ed87edf4626d2024210c0',
            'timestamp': 1494424042,
            'upload_date': '20170510',
        },
        'params': {
            'skip_download': True,
        },
    }, {
        'url': 'https://www.funk.net/channel/59d5149841dca100012511e3/mein-erster-job-lovemilla-folge-1/lovemilla/',
        'only_matching': True,
@ -87,16 +119,56 @@ class FunkChannelIE(FunkBaseIE):
        channel_id = mobj.group('id')
        alias = mobj.group('alias')
-        results = self._download_json(
+        headers = self._make_headers(url)
            'https://www.funk.net/api/v3.0/content/videos/filter', channel_id,
            headers={
                'authorization': 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjbGllbnROYW1lIjoiY3VyYXRpb24tdG9vbCIsInNjb3BlIjoic3RhdGljLWNvbnRlbnQtYXBpLGN1cmF0aW9uLWFwaSxzZWFyY2gtYXBpIn0.q4Y2xZG8PFHai24-4Pjx2gym9RmJejtmK6lMXP5wAgc',
                'Referer': url,
            }, query={
                'channelId': channel_id,
                'size': 100,
            })['result']
-        video = next(r for r in results if r.get('alias') == alias)
+        video = None
        # Id-based channels are currently broken on their side: webplayer
        # tries to process them via byChannelAlias endpoint and fails
        # predictably.
        for page_num in itertools.count():
            by_channel_alias = self._download_json(
                'https://www.funk.net/api/v3.1/webapp/videos/byChannelAlias/%s'
                % channel_id,
                'Downloading byChannelAlias JSON page %d' % (page_num + 1),
                headers=headers, query={
                    'filterFsk': 'false',
                    'sort': 'creationDate,desc',
                    'size': 100,
                    'page': page_num,
                }, fatal=False)
            if not by_channel_alias:
                break
            video_list = try_get(
                by_channel_alias, lambda x: x['_embedded']['videoList'], list)
            if not video_list:
                break
            try:
                video = next(r for r in video_list if r.get('alias') == alias)
                break
            except StopIteration:
                pass
            if not try_get(
                    by_channel_alias, lambda x: x['_links']['next']):
                break
        if not video:
            by_id_list = self._download_json(
                'https://www.funk.net/api/v3.0/content/videos/byIdList',
                channel_id, 'Downloading byIdList JSON', headers=headers,
                query={
                    'ids': alias,
                }, fatal=False)
            if by_id_list:
                video = try_get(by_id_list, lambda x: x['result'][0], dict)
        if not video:
            results = self._download_json(
                'https://www.funk.net/api/v3.0/content/videos/filter',
                channel_id, 'Downloading filter JSON', headers=headers, query={
                    'channelId': channel_id,
                    'size': 100,
                })['result']
            video = next(r for r in results if r.get('alias') == alias)
        return self._make_url_result(video)
--- a/youtube_dl/extractor/fxnetworks.py
+++ b/youtube_dl/extractor/fxnetworks.py
@ -41,7 +41,7 @@ class FXNetworksIE(AdobePassIE):
        if 'The content you are trying to access is not available in your region.' in webpage:
            self.raise_geo_restricted()
        video_data = extract_attributes(self._search_regex(
-            r'(<a.+?rel="http://link\.theplatform\.com/s/.+?</a>)', webpage, 'video data'))
+            r'(<a.+?rel="https?://link\.theplatform\.com/s/.+?</a>)', webpage, 'video data'))
        player_type = self._search_regex(r'playerType\s*=\s*[\'"]([^\'"]+)', webpage, 'player type', default=None)
        release_url = video_data['rel']
        title = video_data['data-title']
--- a/youtube_dl/extractor/gdcvault.py
+++ b/youtube_dl/extractor/gdcvault.py
@ -91,7 +91,7 @@ class GDCVaultIE(InfoExtractor):
    ]
    def _login(self, webpage_url, display_id):
-        (username, password) = self._get_login_info()
+        username, password = self._get_login_info()
        if username is None or password is None:
            self.report_warning('It looks like ' + webpage_url + ' requires a login. Try specifying a username and password and try again.')
            return None
--- a/youtube_dl/extractor/generic.py
+++ b/youtube_dl/extractor/generic.py
@ -23,6 +23,7 @@ from ..utils import (
    is_html,
    js_to_json,
    KNOWN_EXTENSIONS,
    merge_dicts,
    mimetype2ext,
    orderedSet,
    sanitized_Request,
@ -58,6 +59,7 @@ from .xhamster import XHamsterEmbedIE
 from .tnaflix import TNAFlixNetworkEmbedIE
 from .drtuber import DrTuberIE
 from .redtube import RedTubeIE
 from .tube8 import Tube8IE
 from .vimeo import VimeoIE
 from .dailymotion import DailymotionIE
 from .dailymail import DailyMailIE
@ -105,6 +107,13 @@ from .springboardplatform import SpringboardPlatformIE
 from .yapfiles import YapFilesIE
 from .vice import ViceIE
 from .xfileshare import XFileShareIE
 from .cloudflarestream import CloudflareStreamIE
 from .peertube import PeerTubeIE
 from .indavideo import IndavideoEmbedIE
 from .apa import APAIE
 from .foxnews import FoxNewsIE
 from .viqeo import ViqeoIE
 from .expressen import ExpressenIE
 class GenericIE(InfoExtractor):
@ -189,6 +198,16 @@ class GenericIE(InfoExtractor):
                'title': 'pdv_maddow_netcast_m4v-02-27-2015-201624',
            }
        },
        # RSS feed with enclosures and unsupported link URLs
        {
            'url': 'http://www.hellointernet.fm/podcast?format=rss',
            'info_dict': {
                'id': 'http://www.hellointernet.fm/podcast?format=rss',
                'description': 'CGP Grey and Brady Haran talk about YouTube, life, work, whatever.',
                'title': 'Hello Internet',
            },
            'playlist_mincount': 100,
        },
        # SMIL from http://videolectures.net/promogram_igor_mekjavic_eng
        {
            'url': 'http://videolectures.net/promogram_igor_mekjavic_eng/video/1/smil.xml',
@ -1219,7 +1238,7 @@ class GenericIE(InfoExtractor):
                'title': '35871',
                'timestamp': 1355743100,
                'upload_date': '20121217',
-                'uploader_id': 'batchUser',
+                'uploader_id': 'cplapp@learn360.com',
            },
            'add_ie': ['Kaltura'],
        },
@ -1270,6 +1289,39 @@ class GenericIE(InfoExtractor):
            },
            'add_ie': ['Kaltura'],
        },
        {
            # Kaltura iframe embed, more sophisticated
            'url': 'http://www.cns.nyu.edu/~eero/math-tools/Videos/lecture-05sep2017.html',
            'info_dict': {
                'id': '1_9gzouybz',
                'ext': 'mp4',
                'title': 'lecture-05sep2017',
                'description': 'md5:40f347d91fd4ba047e511c5321064b49',
                'upload_date': '20170913',
                'uploader_id': 'eps2',
                'timestamp': 1505340777,
            },
            'params': {
                'skip_download': True,
            },
            'add_ie': ['Kaltura'],
        },
        {
            # meta twitter:player
            'url': 'http://thechive.com/2017/12/08/all-i-want-for-christmas-is-more-twerk/',
            'info_dict': {
                'id': '0_01b42zps',
                'ext': 'mp4',
                'title': 'Main Twerk (Video)',
                'upload_date': '20171208',
                'uploader_id': 'sebastian.salinas@thechive.com',
                'timestamp': 1512713057,
            },
            'params': {
                'skip_download': True,
            },
            'add_ie': ['Kaltura'],
        },
        # referrer protected EaglePlatform embed
        {
            'url': 'https://tvrain.ru/lite/teleshow/kak_vse_nachinalos/namin-418921/',
@ -1345,17 +1397,6 @@ class GenericIE(InfoExtractor):
                'skip_download': True,
            },
        },
        # SVT embed
        {
            'url': 'http://www.svt.se/sport/ishockey/jagr-tacklar-giroux-under-intervjun',
            'info_dict': {
                'id': '2900353',
                'ext': 'flv',
                'title': 'Här trycker Jagr till Giroux (under SVT-intervjun)',
                'duration': 27,
                'age_limit': 0,
            },
        },
        # Crooks and Liars embed
        {
            'url': 'http://crooksandliars.com/2015/04/fox-friends-says-protecting-atheists',
@ -1426,21 +1467,6 @@ class GenericIE(InfoExtractor):
            },
            'expected_warnings': ['Failed to parse JSON Expecting value'],
        },
        # Ooyala embed
        {
            'url': 'http://www.businessinsider.com/excel-index-match-vlookup-video-how-to-2015-2?IR=T',
            'info_dict': {
                'id': '50YnY4czr4ms1vJ7yz3xzq0excz_pUMs',
                'ext': 'mp4',
                'description': 'Index/Match versus VLOOKUP.',
                'title': 'This is what separates the Excel masters from the wannabes',
                'duration': 191.933,
            },
            'params': {
                # m3u8 downloads
                'skip_download': True,
            }
        },
        # Brightcove URL in single quotes
        {
            'url': 'http://www.sportsnet.ca/baseball/mlb/sn-presents-russell-martin-world-citizen/',
@ -1967,7 +1993,98 @@ class GenericIE(InfoExtractor):
            'params': {
                'skip_download': True,
            },
-        }
+        },
        {
            # CloudflareStream embed
            'url': 'https://www.cloudflare.com/products/cloudflare-stream/',
            'info_dict': {
                'id': '31c9291ab41fac05471db4e73aa11717',
                'ext': 'mp4',
                'title': '31c9291ab41fac05471db4e73aa11717',
            },
            'add_ie': [CloudflareStreamIE.ie_key()],
            'params': {
                'skip_download': True,
            },
        },
        {
            # PeerTube embed
            'url': 'https://joinpeertube.org/fr/home/',
            'info_dict': {
                'id': 'home',
                'title': 'Reprenez le contrôle de vos vidéos ! #JoinPeertube',
            },
            'playlist_count': 2,
        },
        {
            # Indavideo embed
            'url': 'https://streetkitchen.hu/receptek/igy_kell_otthon_hamburgert_sutni/',
            'info_dict': {
                'id': '1693903',
                'ext': 'mp4',
                'title': 'Így kell otthon hamburgert sütni',
                'description': 'md5:f5a730ecf900a5c852e1e00540bbb0f7',
                'timestamp': 1426330212,
                'upload_date': '20150314',
                'uploader': 'StreetKitchen',
                'uploader_id': '546363',
            },
            'add_ie': [IndavideoEmbedIE.ie_key()],
            'params': {
                'skip_download': True,
            },
        },
        {
            # APA embed via JWPlatform embed
            'url': 'http://www.vol.at/blue-man-group/5593454',
            'info_dict': {
                'id': 'jjv85FdZ',
                'ext': 'mp4',
                'title': '"Blau ist mysteriös": Die Blue Man Group im Interview',
                'description': 'md5:d41d8cd98f00b204e9800998ecf8427e',
                'thumbnail': r're:^https?://.*\.jpg$',
                'duration': 254,
                'timestamp': 1519211149,
                'upload_date': '20180221',
            },
            'params': {
                'skip_download': True,
            },
        },
        {
            'url': 'http://share-videos.se/auto/video/83645793?uid=13',
            'md5': 'b68d276de422ab07ee1d49388103f457',
            'info_dict': {
                'id': '83645793',
                'title': 'Lock up and get excited',
                'ext': 'mp4'
            },
            'skip': 'TODO: fix nested playlists processing in tests',
        },
        {
            # Viqeo embeds
            'url': 'https://viqeo.tv/',
            'info_dict': {
                'id': 'viqeo',
                'title': 'All-new video platform',
            },
            'playlist_count': 6,
        },
        {
            # videojs embed
            'url': 'https://video.sibnet.ru/shell.php?videoid=3422904',
            'info_dict': {
                'id': 'shell',
                'ext': 'mp4',
                'title': 'Доставщик пиццы спросил разрешения сыграть на фортепиано',
                'description': 'md5:89209cdc587dab1e4a090453dbaa2cb1',
                'thumbnail': r're:^https?://.*\.jpg$',
            },
            'params': {
                'skip_download': True,
            },
            'expected_warnings': ['Failed to download MPD manifest'],
        },
        # {
        #     # TODO: find another test
        #     # http://schema.org/VideoObject
@ -1998,13 +2115,15 @@ class GenericIE(InfoExtractor):
        entries = []
        for it in doc.findall('./channel/item'):
-            next_url = xpath_text(it, 'link', fatal=False)
+            next_url = None
            enclosure_nodes = it.findall('./enclosure')
            for e in enclosure_nodes:
                next_url = e.attrib.get('url')
                if next_url:
                    break
            if not next_url:
-                enclosure_nodes = it.findall('./enclosure')
+                next_url = xpath_text(it, 'link', fatal=False)
                for e in enclosure_nodes:
                    next_url = e.attrib.get('url')
                    if next_url:
                        break
            if not next_url:
                continue
@ -2546,6 +2665,11 @@ class GenericIE(InfoExtractor):
        if redtube_urls:
            return self.playlist_from_matches(redtube_urls, video_id, video_title, ie=RedTubeIE.ie_key())
        # Look for embedded Tube8 player
        tube8_urls = Tube8IE._extract_urls(webpage)
        if tube8_urls:
            return self.playlist_from_matches(tube8_urls, video_id, video_title, ie=Tube8IE.ie_key())
        # Look for embedded Tvigle player
        mobj = re.search(
            r'<iframe[^>]+?src=(["\'])(?P<url>(?:https?:)?//cloud\.tvigle\.ru/video/.+?)\1', webpage)
@ -2899,7 +3023,7 @@ class GenericIE(InfoExtractor):
                wapo_urls, video_id, video_title, ie=WashingtonPostIE.ie_key())
        # Look for Mediaset embeds
-        mediaset_urls = MediasetIE._extract_urls(webpage)
+        mediaset_urls = MediasetIE._extract_urls(self, webpage)
        if mediaset_urls:
            return self.playlist_from_matches(
                mediaset_urls, video_id, video_title, ie=MediasetIE.ie_key())
@ -2963,20 +3087,47 @@ class GenericIE(InfoExtractor):
            return self.playlist_from_matches(
                xfileshare_urls, video_id, video_title, ie=XFileShareIE.ie_key())
-        def merge_dicts(dict1, dict2):
+        cloudflarestream_urls = CloudflareStreamIE._extract_urls(webpage)
-            merged = {}
+        if cloudflarestream_urls:
-            for k, v in dict1.items():
+            return self.playlist_from_matches(
-                if v is not None:
+                cloudflarestream_urls, video_id, video_title, ie=CloudflareStreamIE.ie_key())
-                    merged[k] = v
+
-            for k, v in dict2.items():
+        peertube_urls = PeerTubeIE._extract_urls(webpage, url)
-                if v is None:
+        if peertube_urls:
-                    continue
+            return self.playlist_from_matches(
-                if (k not in merged or
+                peertube_urls, video_id, video_title, ie=PeerTubeIE.ie_key())
-                        (isinstance(v, compat_str) and v and
+
-                            isinstance(merged[k], compat_str) and
+        indavideo_urls = IndavideoEmbedIE._extract_urls(webpage)
-                            not merged[k])):
+        if indavideo_urls:
-                    merged[k] = v
+            return self.playlist_from_matches(
-            return merged
+                indavideo_urls, video_id, video_title, ie=IndavideoEmbedIE.ie_key())
        apa_urls = APAIE._extract_urls(webpage)
        if apa_urls:
            return self.playlist_from_matches(
                apa_urls, video_id, video_title, ie=APAIE.ie_key())
        foxnews_urls = FoxNewsIE._extract_urls(webpage)
        if foxnews_urls:
            return self.playlist_from_matches(
                foxnews_urls, video_id, video_title, ie=FoxNewsIE.ie_key())
        sharevideos_urls = [sharevideos_mobj.group('url') for sharevideos_mobj in re.finditer(
            r'<iframe[^>]+?\bsrc\s*=\s*(["\'])(?P<url>(?:https?:)?//embed\.share-videos\.se/auto/embed/\d+\?.*?\buid=\d+.*?)\1',
            webpage)]
        if sharevideos_urls:
            return self.playlist_from_matches(
                sharevideos_urls, video_id, video_title)
        viqeo_urls = ViqeoIE._extract_urls(webpage)
        if viqeo_urls:
            return self.playlist_from_matches(
                viqeo_urls, video_id, video_title, ie=ViqeoIE.ie_key())
        expressen_urls = ExpressenIE._extract_urls(webpage)
        if expressen_urls:
            return self.playlist_from_matches(
                expressen_urls, video_id, video_title, ie=ExpressenIE.ie_key())
        # Look for HTML5 media
        entries = self._parse_html5_media_entries(url, webpage, video_id, m3u8_id='hls')
@ -2999,9 +3150,13 @@ class GenericIE(InfoExtractor):
        jwplayer_data = self._find_jwplayer_data(
            webpage, video_id, transform_source=js_to_json)
        if jwplayer_data:
-            info = self._parse_jwplayer_data(
+            try:
-                jwplayer_data, video_id, require_title=False, base_url=url)
+                info = self._parse_jwplayer_data(
-            return merge_dicts(info, info_dict)
+                    jwplayer_data, video_id, require_title=False, base_url=url)
                return merge_dicts(info, info_dict)
            except ExtractorError:
                # See https://github.com/rg3/youtube-dl/pull/16735
                pass
        # Video.js embed
        mobj = re.search(
--- a/youtube_dl/extractor/globo.py
+++ b/youtube_dl/extractor/globo.py
@ -1,15 +1,16 @@
 # coding: utf-8
 from __future__ import unicode_literals
 import base64
 import hashlib
 import json
 import random
 import re
 import math
 from .common import InfoExtractor
 from ..compat import (
    compat_HTTPError,
    compat_str,
    compat_chr,
    compat_ord,
 )
 from ..utils import (
    ExtractorError,
@ -22,12 +23,7 @@ from ..utils import (
 class GloboIE(InfoExtractor):
    _VALID_URL = r'(?:globo:|https?://.+?\.globo\.com/(?:[^/]+/)*(?:v/(?:[^/]+/)?|videos/))(?P<id>\d{7,})'
-
+    _NETRC_MACHINE = 'globo'
    _API_URL_TEMPLATE = 'http://api.globovideos.com/videos/%s/playlist'
    _SECURITY_URL_TEMPLATE = 'http://security.video.globo.com/videos/%s/hash?player=flash&version=17.0.0.132&resource_id=%s'
    _RESIGN_EXPIRATION = 86400
    _TESTS = [{
        'url': 'http://g1.globo.com/carros/autoesporte/videos/t/exclusivos-do-g1/v/mercedes-benz-gla-passa-por-teste-de-colisao-na-europa/3607726/',
        'md5': 'b3ccc801f75cd04a914d51dadb83a78d',
@ -70,287 +66,51 @@ class GloboIE(InfoExtractor):
        'only_matching': True,
    }]
-    class MD5(object):
+    def _real_initialize(self):
-        HEX_FORMAT_LOWERCASE = 0
+        email, password = self._get_login_info()
-        HEX_FORMAT_UPPERCASE = 1
+        if email is None:
-        BASE64_PAD_CHARACTER_DEFAULT_COMPLIANCE = ''
+            return
        BASE64_PAD_CHARACTER_RFC_COMPLIANCE = '='
        PADDING = '=0xFF01DD'
        hexcase = 0
        b64pad = ''
-        def __init__(self):
+        try:
-            pass
+            self._download_json(
-
+                'https://login.globo.com/api/authentication', None, data=json.dumps({
-        class JSArray(list):
+                    'payload': {
-            def __getitem__(self, y):
+                        'email': email,
-                try:
+                        'password': password,
-                    return list.__getitem__(self, y)
+                        'serviceId': 4654,
-                except IndexError:
+                    },
-                    return 0
+                }).encode(), headers={
-
+                    'Content-Type': 'application/json; charset=utf-8',
-            def __setitem__(self, i, y):
+                })
-                try:
+        except ExtractorError as e:
-                    return list.__setitem__(self, i, y)
+            if isinstance(e.cause, compat_HTTPError) and e.cause.code == 401:
-                except IndexError:
+                resp = self._parse_json(e.cause.read(), None)
-                    self.extend([0] * (i - len(self) + 1))
+                raise ExtractorError(resp.get('userMessage') or resp['id'], expected=True)
-                    self[-1] = y
+            raise
        @classmethod
        def hex_md5(cls, param1):
            return cls.rstr2hex(cls.rstr_md5(cls.str2rstr_utf8(param1)))
        @classmethod
        def b64_md5(cls, param1, param2=None):
            return cls.rstr2b64(cls.rstr_md5(cls.str2rstr_utf8(param1, param2)))
        @classmethod
        def any_md5(cls, param1, param2):
            return cls.rstr2any(cls.rstr_md5(cls.str2rstr_utf8(param1)), param2)
        @classmethod
        def rstr_md5(cls, param1):
            return cls.binl2rstr(cls.binl_md5(cls.rstr2binl(param1), len(param1) * 8))
        @classmethod
        def rstr2hex(cls, param1):
            _loc_2 = '0123456789ABCDEF' if cls.hexcase else '0123456789abcdef'
            _loc_3 = ''
            for _loc_5 in range(0, len(param1)):
                _loc_4 = compat_ord(param1[_loc_5])
                _loc_3 += _loc_2[_loc_4 >> 4 & 15] + _loc_2[_loc_4 & 15]
            return _loc_3
        @classmethod
        def rstr2b64(cls, param1):
            _loc_2 = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_'
            _loc_3 = ''
            _loc_4 = len(param1)
            for _loc_5 in range(0, _loc_4, 3):
                _loc_6_1 = compat_ord(param1[_loc_5]) << 16
                _loc_6_2 = compat_ord(param1[_loc_5 + 1]) << 8 if _loc_5 + 1 < _loc_4 else 0
                _loc_6_3 = compat_ord(param1[_loc_5 + 2]) if _loc_5 + 2 < _loc_4 else 0
                _loc_6 = _loc_6_1 | _loc_6_2 | _loc_6_3
                for _loc_7 in range(0, 4):
                    if _loc_5 * 8 + _loc_7 * 6 > len(param1) * 8:
                        _loc_3 += cls.b64pad
                    else:
                        _loc_3 += _loc_2[_loc_6 >> 6 * (3 - _loc_7) & 63]
            return _loc_3
        @staticmethod
        def rstr2any(param1, param2):
            _loc_3 = len(param2)
            _loc_4 = []
            _loc_9 = [0] * ((len(param1) >> 2) + 1)
            for _loc_5 in range(0, len(_loc_9)):
                _loc_9[_loc_5] = compat_ord(param1[_loc_5 * 2]) << 8 | compat_ord(param1[_loc_5 * 2 + 1])
            while len(_loc_9) > 0:
                _loc_8 = []
                _loc_7 = 0
                for _loc_5 in range(0, len(_loc_9)):
                    _loc_7 = (_loc_7 << 16) + _loc_9[_loc_5]
                    _loc_6 = math.floor(_loc_7 / _loc_3)
                    _loc_7 -= _loc_6 * _loc_3
                    if len(_loc_8) > 0 or _loc_6 > 0:
                        _loc_8[len(_loc_8)] = _loc_6
                _loc_4[len(_loc_4)] = _loc_7
                _loc_9 = _loc_8
            _loc_10 = ''
            _loc_5 = len(_loc_4) - 1
            while _loc_5 >= 0:
                _loc_10 += param2[_loc_4[_loc_5]]
                _loc_5 -= 1
            return _loc_10
        @classmethod
        def str2rstr_utf8(cls, param1, param2=None):
            _loc_3 = ''
            _loc_4 = -1
            if not param2:
                param2 = cls.PADDING
            param1 = param1 + param2[1:9]
            while True:
                _loc_4 += 1
                if _loc_4 >= len(param1):
                    break
                _loc_5 = compat_ord(param1[_loc_4])
                _loc_6 = compat_ord(param1[_loc_4 + 1]) if _loc_4 + 1 < len(param1) else 0
                if 55296 <= _loc_5 <= 56319 and 56320 <= _loc_6 <= 57343:
                    _loc_5 = 65536 + ((_loc_5 & 1023) << 10) + (_loc_6 & 1023)
                    _loc_4 += 1
                if _loc_5 <= 127:
                    _loc_3 += compat_chr(_loc_5)
                    continue
                if _loc_5 <= 2047:
                    _loc_3 += compat_chr(192 | _loc_5 >> 6 & 31) + compat_chr(128 | _loc_5 & 63)
                    continue
                if _loc_5 <= 65535:
                    _loc_3 += compat_chr(224 | _loc_5 >> 12 & 15) + compat_chr(128 | _loc_5 >> 6 & 63) + compat_chr(
                        128 | _loc_5 & 63)
                    continue
                if _loc_5 <= 2097151:
                    _loc_3 += compat_chr(240 | _loc_5 >> 18 & 7) + compat_chr(128 | _loc_5 >> 12 & 63) + compat_chr(
                        128 | _loc_5 >> 6 & 63) + compat_chr(128 | _loc_5 & 63)
            return _loc_3
        @staticmethod
        def rstr2binl(param1):
            _loc_2 = [0] * ((len(param1) >> 2) + 1)
            for _loc_3 in range(0, len(_loc_2)):
                _loc_2[_loc_3] = 0
            for _loc_3 in range(0, len(param1) * 8, 8):
                _loc_2[_loc_3 >> 5] |= (compat_ord(param1[_loc_3 // 8]) & 255) << _loc_3 % 32
            return _loc_2
        @staticmethod
        def binl2rstr(param1):
            _loc_2 = ''
            for _loc_3 in range(0, len(param1) * 32, 8):
                _loc_2 += compat_chr(param1[_loc_3 >> 5] >> _loc_3 % 32 & 255)
            return _loc_2
        @classmethod
        def binl_md5(cls, param1, param2):
            param1 = cls.JSArray(param1)
            param1[param2 >> 5] |= 128 << param2 % 32
            param1[(param2 + 64 >> 9 << 4) + 14] = param2
            _loc_3 = 1732584193
            _loc_4 = -271733879
            _loc_5 = -1732584194
            _loc_6 = 271733878
            for _loc_7 in range(0, len(param1), 16):
                _loc_8 = _loc_3
                _loc_9 = _loc_4
                _loc_10 = _loc_5
                _loc_11 = _loc_6
                _loc_3 = cls.md5_ff(_loc_3, _loc_4, _loc_5, _loc_6, param1[_loc_7 + 0], 7, -680876936)
                _loc_6 = cls.md5_ff(_loc_6, _loc_3, _loc_4, _loc_5, param1[_loc_7 + 1], 12, -389564586)
                _loc_5 = cls.md5_ff(_loc_5, _loc_6, _loc_3, _loc_4, param1[_loc_7 + 2], 17, 606105819)
                _loc_4 = cls.md5_ff(_loc_4, _loc_5, _loc_6, _loc_3, param1[_loc_7 + 3], 22, -1044525330)
                _loc_3 = cls.md5_ff(_loc_3, _loc_4, _loc_5, _loc_6, param1[_loc_7 + 4], 7, -176418897)
                _loc_6 = cls.md5_ff(_loc_6, _loc_3, _loc_4, _loc_5, param1[_loc_7 + 5], 12, 1200080426)
                _loc_5 = cls.md5_ff(_loc_5, _loc_6, _loc_3, _loc_4, param1[_loc_7 + 6], 17, -1473231341)
                _loc_4 = cls.md5_ff(_loc_4, _loc_5, _loc_6, _loc_3, param1[_loc_7 + 7], 22, -45705983)
                _loc_3 = cls.md5_ff(_loc_3, _loc_4, _loc_5, _loc_6, param1[_loc_7 + 8], 7, 1770035416)
                _loc_6 = cls.md5_ff(_loc_6, _loc_3, _loc_4, _loc_5, param1[_loc_7 + 9], 12, -1958414417)
                _loc_5 = cls.md5_ff(_loc_5, _loc_6, _loc_3, _loc_4, param1[_loc_7 + 10], 17, -42063)
                _loc_4 = cls.md5_ff(_loc_4, _loc_5, _loc_6, _loc_3, param1[_loc_7 + 11], 22, -1990404162)
                _loc_3 = cls.md5_ff(_loc_3, _loc_4, _loc_5, _loc_6, param1[_loc_7 + 12], 7, 1804603682)
                _loc_6 = cls.md5_ff(_loc_6, _loc_3, _loc_4, _loc_5, param1[_loc_7 + 13], 12, -40341101)
                _loc_5 = cls.md5_ff(_loc_5, _loc_6, _loc_3, _loc_4, param1[_loc_7 + 14], 17, -1502002290)
                _loc_4 = cls.md5_ff(_loc_4, _loc_5, _loc_6, _loc_3, param1[_loc_7 + 15], 22, 1236535329)
                _loc_3 = cls.md5_gg(_loc_3, _loc_4, _loc_5, _loc_6, param1[_loc_7 + 1], 5, -165796510)
                _loc_6 = cls.md5_gg(_loc_6, _loc_3, _loc_4, _loc_5, param1[_loc_7 + 6], 9, -1069501632)
                _loc_5 = cls.md5_gg(_loc_5, _loc_6, _loc_3, _loc_4, param1[_loc_7 + 11], 14, 643717713)
                _loc_4 = cls.md5_gg(_loc_4, _loc_5, _loc_6, _loc_3, param1[_loc_7 + 0], 20, -373897302)
                _loc_3 = cls.md5_gg(_loc_3, _loc_4, _loc_5, _loc_6, param1[_loc_7 + 5], 5, -701558691)
                _loc_6 = cls.md5_gg(_loc_6, _loc_3, _loc_4, _loc_5, param1[_loc_7 + 10], 9, 38016083)
                _loc_5 = cls.md5_gg(_loc_5, _loc_6, _loc_3, _loc_4, param1[_loc_7 + 15], 14, -660478335)
                _loc_4 = cls.md5_gg(_loc_4, _loc_5, _loc_6, _loc_3, param1[_loc_7 + 4], 20, -405537848)
                _loc_3 = cls.md5_gg(_loc_3, _loc_4, _loc_5, _loc_6, param1[_loc_7 + 9], 5, 568446438)
                _loc_6 = cls.md5_gg(_loc_6, _loc_3, _loc_4, _loc_5, param1[_loc_7 + 14], 9, -1019803690)
                _loc_5 = cls.md5_gg(_loc_5, _loc_6, _loc_3, _loc_4, param1[_loc_7 + 3], 14, -187363961)
                _loc_4 = cls.md5_gg(_loc_4, _loc_5, _loc_6, _loc_3, param1[_loc_7 + 8], 20, 1163531501)
                _loc_3 = cls.md5_gg(_loc_3, _loc_4, _loc_5, _loc_6, param1[_loc_7 + 13], 5, -1444681467)
                _loc_6 = cls.md5_gg(_loc_6, _loc_3, _loc_4, _loc_5, param1[_loc_7 + 2], 9, -51403784)
                _loc_5 = cls.md5_gg(_loc_5, _loc_6, _loc_3, _loc_4, param1[_loc_7 + 7], 14, 1735328473)
                _loc_4 = cls.md5_gg(_loc_4, _loc_5, _loc_6, _loc_3, param1[_loc_7 + 12], 20, -1926607734)
                _loc_3 = cls.md5_hh(_loc_3, _loc_4, _loc_5, _loc_6, param1[_loc_7 + 5], 4, -378558)
                _loc_6 = cls.md5_hh(_loc_6, _loc_3, _loc_4, _loc_5, param1[_loc_7 + 8], 11, -2022574463)
                _loc_5 = cls.md5_hh(_loc_5, _loc_6, _loc_3, _loc_4, param1[_loc_7 + 11], 16, 1839030562)
                _loc_4 = cls.md5_hh(_loc_4, _loc_5, _loc_6, _loc_3, param1[_loc_7 + 14], 23, -35309556)
                _loc_3 = cls.md5_hh(_loc_3, _loc_4, _loc_5, _loc_6, param1[_loc_7 + 1], 4, -1530992060)
                _loc_6 = cls.md5_hh(_loc_6, _loc_3, _loc_4, _loc_5, param1[_loc_7 + 4], 11, 1272893353)
                _loc_5 = cls.md5_hh(_loc_5, _loc_6, _loc_3, _loc_4, param1[_loc_7 + 7], 16, -155497632)
                _loc_4 = cls.md5_hh(_loc_4, _loc_5, _loc_6, _loc_3, param1[_loc_7 + 10], 23, -1094730640)
                _loc_3 = cls.md5_hh(_loc_3, _loc_4, _loc_5, _loc_6, param1[_loc_7 + 13], 4, 681279174)
                _loc_6 = cls.md5_hh(_loc_6, _loc_3, _loc_4, _loc_5, param1[_loc_7 + 0], 11, -358537222)
                _loc_5 = cls.md5_hh(_loc_5, _loc_6, _loc_3, _loc_4, param1[_loc_7 + 3], 16, -722521979)
                _loc_4 = cls.md5_hh(_loc_4, _loc_5, _loc_6, _loc_3, param1[_loc_7 + 6], 23, 76029189)
                _loc_3 = cls.md5_hh(_loc_3, _loc_4, _loc_5, _loc_6, param1[_loc_7 + 9], 4, -640364487)
                _loc_6 = cls.md5_hh(_loc_6, _loc_3, _loc_4, _loc_5, param1[_loc_7 + 12], 11, -421815835)
                _loc_5 = cls.md5_hh(_loc_5, _loc_6, _loc_3, _loc_4, param1[_loc_7 + 15], 16, 530742520)
                _loc_4 = cls.md5_hh(_loc_4, _loc_5, _loc_6, _loc_3, param1[_loc_7 + 2], 23, -995338651)
                _loc_3 = cls.md5_ii(_loc_3, _loc_4, _loc_5, _loc_6, param1[_loc_7 + 0], 6, -198630844)
                _loc_6 = cls.md5_ii(_loc_6, _loc_3, _loc_4, _loc_5, param1[_loc_7 + 7], 10, 1126891415)
                _loc_5 = cls.md5_ii(_loc_5, _loc_6, _loc_3, _loc_4, param1[_loc_7 + 14], 15, -1416354905)
                _loc_4 = cls.md5_ii(_loc_4, _loc_5, _loc_6, _loc_3, param1[_loc_7 + 5], 21, -57434055)
                _loc_3 = cls.md5_ii(_loc_3, _loc_4, _loc_5, _loc_6, param1[_loc_7 + 12], 6, 1700485571)
                _loc_6 = cls.md5_ii(_loc_6, _loc_3, _loc_4, _loc_5, param1[_loc_7 + 3], 10, -1894986606)
                _loc_5 = cls.md5_ii(_loc_5, _loc_6, _loc_3, _loc_4, param1[_loc_7 + 10], 15, -1051523)
                _loc_4 = cls.md5_ii(_loc_4, _loc_5, _loc_6, _loc_3, param1[_loc_7 + 1], 21, -2054922799)
                _loc_3 = cls.md5_ii(_loc_3, _loc_4, _loc_5, _loc_6, param1[_loc_7 + 8], 6, 1873313359)
                _loc_6 = cls.md5_ii(_loc_6, _loc_3, _loc_4, _loc_5, param1[_loc_7 + 15], 10, -30611744)
                _loc_5 = cls.md5_ii(_loc_5, _loc_6, _loc_3, _loc_4, param1[_loc_7 + 6], 15, -1560198380)
                _loc_4 = cls.md5_ii(_loc_4, _loc_5, _loc_6, _loc_3, param1[_loc_7 + 13], 21, 1309151649)
                _loc_3 = cls.md5_ii(_loc_3, _loc_4, _loc_5, _loc_6, param1[_loc_7 + 4], 6, -145523070)
                _loc_6 = cls.md5_ii(_loc_6, _loc_3, _loc_4, _loc_5, param1[_loc_7 + 11], 10, -1120210379)
                _loc_5 = cls.md5_ii(_loc_5, _loc_6, _loc_3, _loc_4, param1[_loc_7 + 2], 15, 718787259)
                _loc_4 = cls.md5_ii(_loc_4, _loc_5, _loc_6, _loc_3, param1[_loc_7 + 9], 21, -343485551)
                _loc_3 = cls.safe_add(_loc_3, _loc_8)
                _loc_4 = cls.safe_add(_loc_4, _loc_9)
                _loc_5 = cls.safe_add(_loc_5, _loc_10)
                _loc_6 = cls.safe_add(_loc_6, _loc_11)
            return [_loc_3, _loc_4, _loc_5, _loc_6]
        @classmethod
        def md5_cmn(cls, param1, param2, param3, param4, param5, param6):
            return cls.safe_add(
                cls.bit_rol(cls.safe_add(cls.safe_add(param2, param1), cls.safe_add(param4, param6)), param5), param3)
        @classmethod
        def md5_ff(cls, param1, param2, param3, param4, param5, param6, param7):
            return cls.md5_cmn(param2 & param3 | ~param2 & param4, param1, param2, param5, param6, param7)
        @classmethod
        def md5_gg(cls, param1, param2, param3, param4, param5, param6, param7):
            return cls.md5_cmn(param2 & param4 | param3 & ~param4, param1, param2, param5, param6, param7)
        @classmethod
        def md5_hh(cls, param1, param2, param3, param4, param5, param6, param7):
            return cls.md5_cmn(param2 ^ param3 ^ param4, param1, param2, param5, param6, param7)
        @classmethod
        def md5_ii(cls, param1, param2, param3, param4, param5, param6, param7):
            return cls.md5_cmn(param3 ^ (param2 | ~param4), param1, param2, param5, param6, param7)
        @classmethod
        def safe_add(cls, param1, param2):
            _loc_3 = (param1 & 65535) + (param2 & 65535)
            _loc_4 = (param1 >> 16) + (param2 >> 16) + (_loc_3 >> 16)
            return cls.lshift(_loc_4, 16) | _loc_3 & 65535
        @classmethod
        def bit_rol(cls, param1, param2):
            return cls.lshift(param1, param2) | (param1 & 0xFFFFFFFF) >> (32 - param2)
        @staticmethod
        def lshift(value, count):
            r = (0xFFFFFFFF & value) << count
            return -(~(r - 1) & 0xFFFFFFFF) if r > 0x7FFFFFFF else r
    def _real_extract(self, url):
        video_id = self._match_id(url)
        video = self._download_json(
-            self._API_URL_TEMPLATE % video_id, video_id)['videos'][0]
+            'http://api.globovideos.com/videos/%s/playlist' % video_id,
            video_id)['videos'][0]
        title = video['title']
        formats = []
        for resource in video['resources']:
            resource_id = resource.get('_id')
-            if not resource_id or resource_id.endswith('manifest'):
+            resource_url = resource.get('url')
            if not resource_id or not resource_url:
                continue
            security = self._download_json(
-                self._SECURITY_URL_TEMPLATE % (video_id, resource_id),
+                'http://security.video.globo.com/videos/%s/hash' % video_id,
-                video_id, 'Downloading security hash for %s' % resource_id)
+                video_id, 'Downloading security hash for %s' % resource_id, query={
                    'player': 'flash',
                    'version': '17.0.0.132',
                    'resource_id': resource_id,
                })
            security_hash = security.get('hash')
            if not security_hash:
@ -361,22 +121,28 @@ class GloboIE(InfoExtractor):
                continue
            hash_code = security_hash[:2]
-            received_time = int(security_hash[2:12])
+            received_time = security_hash[2:12]
            received_random = security_hash[12:22]
            received_md5 = security_hash[22:]
-            sign_time = received_time + self._RESIGN_EXPIRATION
+            sign_time = compat_str(int(received_time) + 86400)
            padding = '%010d' % random.randint(1, 10000000000)
-            signed_md5 = self.MD5.b64_md5(received_md5 + compat_str(sign_time) + padding)
+            md5_data = (received_md5 + sign_time + padding + '0xFF01DD').encode()
-            signed_hash = hash_code + compat_str(received_time) + received_random + compat_str(sign_time) + padding + signed_md5
+            signed_md5 = base64.urlsafe_b64encode(hashlib.md5(md5_data).digest()).decode().strip('=')
            signed_hash = hash_code + received_time + received_random + sign_time + padding + signed_md5
            resource_url = resource['url']
            signed_url = '%s?h=%s&k=%s' % (resource_url, signed_hash, 'flash')
            if resource_id.endswith('m3u8') or resource_url.endswith('.m3u8'):
                formats.extend(self._extract_m3u8_formats(
                    signed_url, resource_id, 'mp4', entry_protocol='m3u8_native',
                    m3u8_id='hls', fatal=False))
            elif resource_id.endswith('mpd') or resource_url.endswith('.mpd'):
                formats.extend(self._extract_mpd_formats(
                    signed_url, resource_id, mpd_id='dash', fatal=False))
            elif resource_id.endswith('manifest') or resource_url.endswith('/manifest'):
                formats.extend(self._extract_ism_formats(
                    signed_url, resource_id, ism_id='mss', fatal=False))
            else:
                formats.append({
                    'url': signed_url,
--- a/youtube_dl/extractor/go.py
+++ b/youtube_dl/extractor/go.py
@ -36,7 +36,8 @@ class GoIE(AdobePassIE):
            'requestor_id': 'DisneyXD',
        }
    }
-    _VALID_URL = r'https?://(?:(?P<sub_domain>%s)\.)?go\.com/(?:(?:[^/]+/)*(?P<id>vdka\w+)|(?:[^/]+/)*(?P<display_id>[^/?#]+))' % '|'.join(_SITE_INFO.keys())
+    _VALID_URL = r'https?://(?:(?P<sub_domain>%s)\.)?go\.com/(?:(?:[^/]+/)*(?P<id>vdka\w+)|(?:[^/]+/)*(?P<display_id>[^/?#]+))'\
                 % '|'.join(list(_SITE_INFO.keys()) + ['disneynow'])
    _TESTS = [{
        'url': 'http://abc.go.com/shows/designated-survivor/video/most-recent/VDKA3807643',
        'info_dict': {
@ -62,6 +63,14 @@ class GoIE(AdobePassIE):
    }, {
        'url': 'http://abc.go.com/shows/world-news-tonight/episode-guide/2017-02/17-021717-intense-stand-off-between-man-with-rifle-and-police-in-oakland',
        'only_matching': True,
    }, {
        # brand 004
        'url': 'http://disneynow.go.com/shows/big-hero-6-the-series/season-01/episode-10-mr-sparkles-loses-his-sparkle/vdka4637915',
        'only_matching': True,
    }, {
        # brand 008
        'url': 'http://disneynow.go.com/shows/minnies-bow-toons/video/happy-campers/vdka4872013',
        'only_matching': True,
    }]
    def _extract_videos(self, brand, video_id='-1', show_id='-1'):
@ -72,14 +81,23 @@ class GoIE(AdobePassIE):
    def _real_extract(self, url):
        sub_domain, video_id, display_id = re.match(self._VALID_URL, url).groups()
-        site_info = self._SITE_INFO[sub_domain]
+        site_info = self._SITE_INFO.get(sub_domain, {})
-        brand = site_info['brand']
+        brand = site_info.get('brand')
-        if not video_id:
+        if not video_id or not site_info:
-            webpage = self._download_webpage(url, display_id)
+            webpage = self._download_webpage(url, display_id or video_id)
            video_id = self._search_regex(
                # There may be inner quotes, e.g. data-video-id="'VDKA3609139'"
                # from http://freeform.go.com/shows/shadowhunters/episodes/season-2/1-this-guilty-blood
-                r'data-video-id=["\']*(VDKA\w+)', webpage, 'video id', default=None)
+                r'data-video-id=["\']*(VDKA\w+)', webpage, 'video id',
                default=None)
            if not site_info:
                brand = self._search_regex(
                    (r'data-brand=\s*["\']\s*(\d+)',
                     r'data-page-brand=\s*["\']\s*(\d+)'), webpage, 'brand',
                    default='004')
                site_info = next(
                    si for _, si in self._SITE_INFO.items()
                    if si.get('brand') == brand)
            if not video_id:
                # show extraction works for Disney, DisneyJunior and DisneyXD
                # ABC and Freeform has different layout
@ -123,7 +141,7 @@ class GoIE(AdobePassIE):
                        'adobe_requestor_id': requestor_id,
                    })
                else:
-                    self._initialize_geo_bypass(['US'])
+                    self._initialize_geo_bypass({'countries': ['US']})
                entitlement = self._download_json(
                    'https://api.entitlement.watchabc.go.com/vp2/ws-secure/entitlement/2020/authorize.json',
                    video_id, data=urlencode_postdata(data))
--- a/youtube_dl/extractor/go90.py
+++ b/youtube_dl/extractor/go90.py
@ -4,16 +4,19 @@ from __future__ import unicode_literals
 import re
 from .common import InfoExtractor
 from ..compat import compat_HTTPError
 from ..utils import (
    determine_ext,
    ExtractorError,
    int_or_none,
    parse_age_limit,
    parse_iso8601,
 )
 class Go90IE(InfoExtractor):
-    _VALID_URL = r'https?://(?:www\.)?go90\.com/videos/(?P<id>[0-9a-zA-Z]+)'
+    _VALID_URL = r'https?://(?:www\.)?go90\.com/(?:videos|embed)/(?P<id>[0-9a-zA-Z]+)'
-    _TEST = {
+    _TESTS = [{
        'url': 'https://www.go90.com/videos/84BUqjLpf9D',
        'md5': 'efa7670dbbbf21a7b07b360652b24a32',
        'info_dict': {
@ -23,16 +26,35 @@ class Go90IE(InfoExtractor):
            'description': 'VICE\'s Karley Sciortino meets with activists who discuss the state\'s strong anti-porn stance. Then, VICE Sports explains NFL contracts.',
            'timestamp': 1491868800,
            'upload_date': '20170411',
            'age_limit': 14,
        }
-    }
+    }, {
        'url': 'https://www.go90.com/embed/261MflWkD3N',
        'only_matching': True,
    }]
    _GEO_BYPASS = False
    def _real_extract(self, url):
        video_id = self._match_id(url)
-        video_data = self._download_json(
+
-            'https://www.go90.com/api/view/items/' + video_id,
+        try:
-            video_id, headers={
+            headers = self.geo_verification_headers()
            headers.update({
                'Content-Type': 'application/json; charset=utf-8',
-            }, data=b'{"client":"web","device_type":"pc"}')
+            })
            video_data = self._download_json(
                'https://www.go90.com/api/view/items/' + video_id, video_id,
                headers=headers, data=b'{"client":"web","device_type":"pc"}')
        except ExtractorError as e:
            if isinstance(e.cause, compat_HTTPError) and e.cause.code == 400:
                message = self._parse_json(e.cause.read().decode(), None)['error']['message']
                if 'region unavailable' in message:
                    self.raise_geo_restricted(countries=['US'])
                raise ExtractorError(message, expected=True)
            raise
        if video_data.get('requires_drm'):
            raise ExtractorError('This video is DRM protected.', expected=True)
        main_video_asset = video_data['main_video_asset']
        episode_number = int_or_none(video_data.get('episode_number'))
@ -123,4 +145,5 @@ class Go90IE(InfoExtractor):
            'season_number': season_number,
            'episode_number': episode_number,
            'subtitles': subtitles,
            'age_limit': parse_age_limit(video_data.get('rating')),
        }
--- a/youtube_dl/extractor/hidive.py
+++ b/youtube_dl/extractor/hidive.py
@ -8,6 +8,7 @@ from ..compat import compat_str
 from ..utils import (
    ExtractorError,
    int_or_none,
    url_or_none,
    urlencode_postdata,
 )
@ -17,6 +18,8 @@ class HiDiveIE(InfoExtractor):
    # Using X-Forwarded-For results in 403 HTTP error for HLS fragments,
    # so disabling geo bypass completely
    _GEO_BYPASS = False
    _NETRC_MACHINE = 'hidive'
    _LOGIN_URL = 'https://www.hidive.com/account/login'
    _TESTS = [{
        'url': 'https://www.hidive.com/stream/the-comic-artist-and-his-assistants/s01e001',
@ -31,8 +34,26 @@ class HiDiveIE(InfoExtractor):
        'params': {
            'skip_download': True,
        },
        'skip': 'Requires Authentication',
    }]
    def _real_initialize(self):
        email, password = self._get_login_info()
        if email is None:
            return
        webpage = self._download_webpage(self._LOGIN_URL, None)
        form = self._search_regex(
            r'(?s)<form[^>]+action="/account/login"[^>]*>(.+?)</form>',
            webpage, 'login form')
        data = self._hidden_inputs(form)
        data.update({
            'Email': email,
            'Password': password,
        })
        self._download_webpage(
            self._LOGIN_URL, None, 'Logging in', data=urlencode_postdata(data))
    def _real_extract(self, url):
        mobj = re.match(self._VALID_URL, url)
        title, key = mobj.group('title', 'key')
@ -43,6 +64,7 @@ class HiDiveIE(InfoExtractor):
            data=urlencode_postdata({
                'Title': title,
                'Key': key,
                'PlayerId': 'f4f895ce1ca713ba263b91caeb1daa2d08904783',
            }))
        restriction = settings.get('restrictionReason')
@ -59,8 +81,8 @@ class HiDiveIE(InfoExtractor):
            bitrates = rendition.get('bitrates')
            if not isinstance(bitrates, dict):
                continue
-            m3u8_url = bitrates.get('hls')
+            m3u8_url = url_or_none(bitrates.get('hls'))
-            if not isinstance(m3u8_url, compat_str):
+            if not m3u8_url:
                continue
            formats.extend(self._extract_m3u8_formats(
                m3u8_url, video_id, 'mp4', entry_protocol='m3u8_native',
@ -72,13 +94,13 @@ class HiDiveIE(InfoExtractor):
                if not isinstance(cc_file, list) or len(cc_file) < 3:
                    continue
                cc_lang = cc_file[0]
-                cc_url = cc_file[2]
+                cc_url = url_or_none(cc_file[2])
-                if not isinstance(cc_lang, compat_str) or not isinstance(
+                if not isinstance(cc_lang, compat_str) or not cc_url:
                        cc_url, compat_str):
                    continue
                subtitles.setdefault(cc_lang, []).append({
                    'url': cc_url,
                })
        self._sort_formats(formats)
        season_number = int_or_none(self._search_regex(
            r's(\d+)', key, 'season number', default=None))
--- a/Show More
+++ b/Show More