python 爬取网上数据Crawler data

1.网页视频直接获取:
网页元素检查:


获取代码如下:

import requests 
from bs4 import BeautifulSoup
from selenium import webdriver  #运用这个模块抓取网页元素很重要

HEADERS = {
    'Accept': 'application/json, text/javascript, */*; q=0.01',
    'referer': 'https://www.pearvideo.com/',
    'x-csrf-token': '',
    'x-requested-with': 'XMLHttpRequest',
    'cookie': ''
    ,
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko)'
                  ' Chrome/90.0.4430.212 Safari/537.36'}

#抓取对象
POPULAR_URL = "https://www.pearvideo.com/video_1761020"
browser = webdriver.Opera()
#获取网页元素:
browser.get(POPULAR_URL)
soup = BeautifulSoup(browser.page_source, 'lxml')
#找到抓取数据位置:
texts = soup.find_all('video')
print(sp1)
输出:
[<video autoplay="autoplay" playsinline="" src="https://video.pearvideo.com/mp4/third/20220505/cont-1761020-15454898-113859-hd.mp4" style="width: 100%; height: 100%;" webkit-playsinline="" x-webkit-airplay=""></video>]
for text in texts:
    MP4_url = text.get('src')  
    print(MP4_url)
    输出:
    https://video.pearvideo.com/mp4/third/20220505/cont-1761020-15454898-113859-hd.mp4
    mp4 = requests.get(MP4_url,headers = HEADERS).content
    with open('视频\%s'%(MP4_url.strip().split('-')[-1]),'wb') as f:
        f.write(mp4)
        f.close()

2.动态网页视频获取:

解析网页,真实视频地址: