EvilSaint Wordmark

Making A YouTube Scraper

25 Aug 2021, 10:45 p.m.
3 Dec 2021, 10:31 a.m.
08:14 minutes

    Making A YouTube Scraper

    PowerShell

    What do we need:

    1. Get an API Credential for Youtube
    2. Query the data from Google API

    Get API Credential for YouTube

    Step 1

    First navigate to the following page to setup the Google API https://console.developers.google.com/apis/dashboard

    Step 2

    Create a new project

    Setting Up a New Project
    Step 3

    Then click on Enable APIs and Services. We will need to search for the Youtube API to specify the scope of access.

    Enabling APIs and Services
    Step 4

    Search for Youtube and select Youtube Data API v3, then select ENABLE.

    You Need to Enable API Access in Order to Use it
    Step 5

    Finally, click on Credentials on the left side, then select Create Credential

    Select Credentials From The Left Sidebar
    Select Create Credentials to Generate New Credentials For The Project
    Step 6

    We are presented with two choices. If in doubt you can select “Help me choose”

    Create Credentials Dropdown
    Step 7

    In this tutorial we will be scraping public data so select that radio option.

    Hit Next to Configure Public Data Access

    After hitting ‘Next’ you should be presented with the following screen.

    Once
    Step 8

    Click on done to be returned to the dashboard.

    Copy Google API Credentials From The Dashboard
    Step 9

    Select your API Key and open up PowerShell

    Querying data from the YouTube API

    You can find more information on the YouTube API at the following URL: https://developers.google.com/youtube/v3/docs/

    Step 1

    In the first example we will use gather a list of videos for the the EminemVEVO user. In order to get a list of videos we first need a channel ID which we can search for by username. Once we have the channel ID and the number of videos on the channel we can request a list of videos.

    Using PowerShell ISE to Query The EminemVEVO YouTube Channel

    Here is the example code.

    ```bash
# Get the API Key
$GoogleApiKey = 'REDACTED'

# Query Google API for the details of the EminemVEVO channel
$data = Invoke-RestMethod -uri "https://youtube.googleapis.com/youtube/v3/channels?part=snippet&part=statistics&part=status&forUsername=EminemVEVO&key=$GoogleApiKey" | Select-Object -expand items

$channelId = $data.id
$maxResults = $data.statistics.videoCount

# Now we can use the search method to search for all the eminem videos in this channel
$eminem = Invoke-RestMethod -uri "https://www.googleapis.com/youtube/v3/search?channelId=$channelId&order=date&part=snippet&type=video&maxResults=$maxResults&key=$GoogleApiKey" | Select-Object -expand items

# List all eminem videos by title
$eminem | ForEach-Object { $_.snippet.title }
```

    The output of this command gives us the following.

    ```bash
Eminem - Killer (Remix) [Official Audio] ft. Jack Harlow, Cordae
Eminem - Alfred's Theme (Lyric Video)
Eminem - Godzilla (Lyric Video) ft. Juice WRLD
Eminem - Darkness (Official Video)
Eminem - Good Guy (Behind The Scenes) ft. Jessie Reyez
Eminem - Good Guy ft. Jessie Reyez
Eminem - Venom
Eminem - Lucky You ft. Joyner Lucas
Eminem - Fall
Eminem - Framed
Eminem - Nowhere Fast (Extended Version) [Audio] ft. Kehlani
Eminem - River (Behind the Scenes) ft. Ed Sheeran
Eminem - Untouchable (Lyric Video)
Eminem - River ft. Ed Sheeran (Official Video)
Eminem - River (Trailer: Boxing) ft. Ed Sheeran
Eminem - Walk On Water ft. Beyoncé (Official Behind The Scenes Video)
Eminem - Walk On Water (Official Video)
Eminem - River (Audio) ft. Ed Sheeran
Eminem ft. Beyoncé - Walk On Water (Official Lyric Video)
Eminem - Untouchable (Audio)
Walk On Water/Stan/Love The Way You Lie (Medley/Live From Saturday Night Live/2017)
Eminem - Walk On Water (Audio) ft. Beyoncé
Eminem - Partners In Rhyme: The True Story of Infinite (Official Trailer)
Eminem - Infinite (F.B.T. Remix) (Official Audio)
Eminem - Phenomenal (Behind The Scenes)
Eminem - Phenomenal
Eminem - Kings Never Die (Lyric Video) ft. Gwen Stefani
Eminem - Kings Never Die (Audio) ft. Gwen Stefani
Eminem - Phenomenal (Lyric Video)
Detroit Vs. Everybody (Behind The Scenes)
Detroit Vs. Everybody (Lyric Video)
Eminem, Royce da 5'9", Big Sean, Danny Brown, Dej Loaf, Trick Trick - Detroit Vs. Everybody
Eminem - Eminem Interview (CD:UK)
Eminem, Slaughterhouse, Yelawolf - CXVPHER (Behind The Scenes)
Eminem - Guts Over Fear ft. Sia (Official Video)
Vevo Presents: Shady CXVPHER (Official Video)
Detroit Vs. Everybody
Vevo Presents: SHADY CXVPHER
Eminem - Guts Over Fear ft. Sia (Lyric Video)
Eminem - Guts Over Fear (Audio) ft. Sia
Eminem - Headlights ft. Nate Ruess (Official Music Video)
Eminem - The Monster Explained (Behind The Scenes) ft. Rihanna
Eminem - The Monster (Edited) ft. Rihanna
Eminem ft. Rihanna - The Monster (Explicit) [Official Video]
Eminem - The Monster (Teaser) ft. Rihanna
Eminem - Rap God (Explicit)
Eminem - Survival (Live on SNL)
Eminem - Berzerk (Live on SNL)
Eminem - The Monster ft. Rihanna (Audio)
Eminem - Survival (Explicit)
```
    Step 2

    I am aware Eminem might not to be everyone’s taste so in this second example we will list ippsec videos. Here we have the channel ID hardcoded.

    Using PowerShell ISE to Query The IppSec YouTube Channel

    The example code

    # Get the API Key
$GoogleApiKey = 'AIzaSyCAhq9IRVmDccixZlOvQ9FFYAWx7Hx3yvY'

$channelId = 'UCa6eh7gCkpPo5XXUDfygQQA'
$maxResults = 100

# Now we can use the search method to search for all the eminem videos in this channel
$ippsec = Invoke-RestMethod -uri "https://www.googleapis.com/youtube/v3/search?channelId=$channelId&order=date&part=snippet&type=video&maxResults=$maxResults&key=$GoogleApiKey" | Select-Object -expand items

# List all eminem videos by title
$ippsec | ForEach-Object { $_.snippet.title }

    This gives us the following output.

    HackTheBox - Proper
HackTheBox - Crossfit2
HackTheBox - Love
HackTheBox - TheNotebook
HackTheBox - Armageddon
HackTheBox - Breadcrumbs
HackTheBox - Atom
HackTheBox - Ophiuchi
HackTheBox - Spectra
HackTheBox - Tentacle
HackTheBox - Tenet
HackTheBox - ScriptKiddie
HackTheBox - Cereal
HackTheBox - Delivery
HackTheBox - Ready
HackTheBox - Attended
HackTheBox - Sharp
HackTheBox - Bucket
HackTheBox - Laboratory
HackTheBox - APT
HackTheBox - Time
HackTheBox - Luanne
HackTheBox - Crossfit
HackTheBox - Reel2
HackTheBox - Passage
HackTheBox - Academy
HackTheBox - Feline
HackTheBox - Jewel
HackTheBox - Doctor
HackTheBox - Worker
HackTheBox - Compromised
HackTheBox - Rope2
HackTheBox - Omni
HackTheBox - Laser
HackTheBox - OpenKeyS
HackTheBox - Unbalanced
HackTheBox - SneakyMailer
HackTheBox - Buff
HackTheBox - Intense
HackTheBox - Academy Intro
HackTheBox - Tabby
HackTheBox - Fuse
HackTheBox - Dyplesher
HackTheBox - Blunder
HackTheBox - Cache
HackTheBox - Blackfield
HackTheBox - Admirer
HackTheBox - Multimaster
HackTheBox - Travel
HackTheBox - Remote
    Linux File Formatting
    Installing Oracle Tools For Pentesting

    Captcha: What's the standard TCP port of the following service?

    captcha

    0 comments