[Python] About crawling

0. How I came across crawling

Although it is difficult to call it development, I learned about crawling because I wanted to know about crawling, which is a way to retrieve existing data in an easy-to-view manner. Of course, I found out about 1차원적인 생각이라는 것 through a search, but since basic learning is also necessary, I decided to use it only for personal learning without infringing on copyright.

1. What is crawling?

Crawling can be said to be a method of organizing information in searches. If there is a word you searched for in 구글 웹 색인, check the URL, the article page, etc. to see where the word is included. It is also used to analyze the reliability of web pages and derive search results according to the search algorithm. Search for pages linked from that web page, enter the hyperlink, and expand the search scope through consecutive actions.

2. Crawling precautions

As can be seen in the Saramin and Job Korea crawling incidents, crawling to the extent of infringing on the copyright of information on a web page is problematic. There is a protocol for controlling access to web crawler robots using robot.txt, which can be viewed by entering the site's top path/robot.txt. Therefore, if there is a part you want to crawl, although it is a recommendation, it is advisable to check and crawl it. Basically, Naver disallows everything, and Google allows some things, so let's check.

3. Import the desired code within one website

You can use querySelectorAll in the console on a web page to derive the value of the desired code. Although it is somewhat different from the search algorithm, let's focus on the value derived. Let's try crawling, which is commonly referred to when searching for web crawler. This is also different from the algorithm for searching through multiple hyperlinks. But, practice getting the code?! I think it would be good to just think about it that way.

It can be displayed using python in VS Code. Before that, open a prompt window for the module and pip install bs4 requests

Download 2 modules.

beautifulSoup is a Python package for parsing HTML and XML. Parsing refers to the process of loading sentences into interpretable units, processing the necessary data on the web, and loading them to the server. requests is an HTTP library for the python programming language. It is also shown in the Wikipedia example, but when loading the page code, you can get it like reqeusts.get("url address").text.

+ Pylance report missing module source

If the Pylance extension recognizes a different version of Python in VS Code, the message “missing module” appears and import fails. This can be resolved by entering python using the Ctrl+Shift+P shortcut keys and selecting the python version that should be applied. In some cases, where re-launching was not possible, re-launching VS Code was sufficient.

+ If the terminal window shows that there is no module

There were cases where the terminal window showed that there were no installed modules. In this case, there are cases where the installation in the corresponding path is not reflected well.

(1) Uninstall and reinstall again

(2) This was resolved by installing from the location where pip.exe is located. The location of pip.exe is in the sub-Scripts folder in the path where python is installed.

4. Create a Telegram bot

Learning how to create a Telegram bot is an essential part of crawling. You can search for BotFather on Telegram and follow the manual to set up bot creation with /newbot. Done! Congratulations on your new bot. When the phrase appears, the token value appears.

After installing with pip install python-telegram-bot,

import telegram

Now you can use the property called Bot, and just enter the token value in parentheses as the token value you received after setting it earlier.

You can send content to the Telegram bot channel using sendMessage() with the above value. In the parentheses, just write the channel id value and the content you want to transmit. The channel id value can be obtained indirectly using getUpdates.

웹 크롤링 어디까지 허용되는가? (Web Crawling, Web Scrapping Issue)

? '웹 크롤러' 좀 그만 만들어라

Google 검색의 원리 | 크롤링 및 색인 생성

No Module Named bs4 beautifulsoup python 파이썬 :: studyhard24

20초도 안걸리는 파이썬 설치 경로 알아내기 !! | 파이썬 설치 경로 확인

Visual Studio Code 에서 Python 패키지의 Unresolved Import Error 를 처리하는 방법

OKKY - 파이썬 pip 및 텔레그램 모듈 설치관련 질문입니다.

Beautiful Soup (HTML parser) - Wikipedia

1. Parsing이란 무엇인가?

Comments