What you'll learn:
- LXML core fundamentals
- XPath & CSS selectors
- How send HTTP requests with Python
- Scraping HTML web pages
- Scraping multiple pages using recursion
- Scraping APIs
- Splash HTTP API
- Scraping JavaScript websites using Splash
- Authentication and Login to websites using Requests
- Web scraping best practices
- Building datasets
What is web scraping ?
Let's say your boss at work gave you a task where he wants you to extract about 1000product from a website, structure the data and save it to a database, would you copy paste manually all the product details from the product name, url and price ? I can imagine you would work days and nights and you wont finish the task, so this is where web scraping shines. So web scraping, or web harvesting or web data extraction is like writing a script that will automate data extraction from websites in a matter of minutes !.
Why learn Web Scraping ?
Whether you're a data analyst, a web developer or even someone who wants to work as a freelancer you should learn web scraping.
For a data analyst building a dataset is extremely important, so without web scraping you simply can't generate it in addition to that adding web scraping in your resume is a plus for you.
Web scraping can be used in a variety of fields, so let me give you some examples on what you can do with it:
Generate leads,
Drop shipping where basically you gonna constantly scrape products from different online stores and show case them on your website to make money,
Monitor products prices to get the best deals,
automation,
Machine learning,
Web scraping freelancer
Of course there are tons and tons and variety of fields where web scraping can be extremely beneficial.
Is this course the right one for you ?
I've carefully planned and designed this course to be beginner friendly, from my experience Iknow those who do web scraping are mostly data analyst with no background knowledge on how the web works, how requests are made, how to locate and parse the data from the web and much much more, in addition to that this the most updated course regarding the material included and the tools used, so in this course
I'll introduce to you the most used web scraping tools/frameworks
We will setup the development environment from scratch
You will learn and understand LXMLcore fundamentals
How to use XPath & CSSselectors to select the data from a web page
How the web works (Request/Response)
How to scrape simple HTMLweb pages
How to scrape multiple web pages
Extract data from APIs
You will learn Splash(crash course) so you can use it to scrape JavaScript websites
Authentication/Login
Store the extracted data whether to JSON/CSVfiles or MongoDb/SQLite3
Exclusive tips and tricks regarding web scraping
Finally this course is project based, each section starting from the 2nd one we will experiment with a different website, each project has a certain degree of difficulty and each one is completely independent from other projects.
Is there is any assignments/exercises included in this course ?
Yes, each section has an assignment included to it, this will help to get your hands dirty and by the end of each section after doing the assignment included you will feel more confident and comfortable with web scraping.
Why LXML and not BeautifulSoup ?
LXMLis a lightweight HTMLparser even the most popular web scraping framework (Scrapy) is built on the top of LXML, BeautifulSoup is a little bit overloaded with the number of functions exposed to us, it has more functions to use, yes that's right ! however in Web Scraping most of the time we use XPath and CSSSelectors to navigate and select what to scrape from the HTMLweb page (tree) so there is no need to learn about new functions and wasting all that time to familiarize yourself with the BeautifulSoup API and the internal architecture, in addition to all of that LXMLin terms of performance is way better than BeautifulSoup.
Who is your instructor ?
Hi! I'm Ahmed nice to meet you, my students prefer to call me web scraping Ninja and currently Ihave taught more than 2000students around the world how to do web scraping. I personallydo web scraping on daily basis whether for fun, for personal projects or as a freelancer and guess what ?Ieven have a master degree in computer science.
Should Ienroll to this course ?
Honestly, by enrolling to this course you have nothing to lose, because if this course didn't meet your requirements, you can always ask for a refund in less than 30days from the day you enrolled to the course guaranteed by Udemy with NO QUESTIONTOASK !
SOIFYOU DON'TKNOWANYTHINGABOUTWEBSCRAPING & YOU DON'T KNOWWHERETOSTARTENROLLNOW ! :)
