[Python] About crawling
0. How I came across crawling
Although it is difficult to call it development, I learned about crawling because I wanted to know about crawling, which is a way to retrieve existing data in an easy-to-view manner. Of course, I found out about 1차원적인 생각이라는 것 through a search, but since basic learning is also necessary, I decided to use it only for personal learning without infringing on copyright.
1. What is crawling?
Crawling can be said to be a method of organizing information in searches. If there is a word you searched for in 구글 웹 색인, check the URL, the article page, etc. to see where the word is included. It is also used to analyze the reliability of web pages and derive search results according to the search algorithm. Search for pages linked from that web page, enter the hyperlink, and expand the search scope through consecutive actions.
2. Crawling precautions
As can be seen in the Saramin and Job Korea crawling incidents, crawling to the extent of infringing on the copyright of information on a web page is problematic. There is a protocol for controlling access to web crawler robots using robot.txt, which can be viewed by entering the site's top path/robot.txt. Therefore, if there is a part you want to crawl, although it is a recommendation, it is advisable to check and crawl it. Basically, Naver disallows everything, and Google allows some things, so let's check.
3. Import the desired code within one website
You can use querySelectorAll in the console on a web page to derive the value of the desired code. Although it is somewhat different from the search algorithm, let's focus on the value derived. Let's try crawling, which is commonly referred to when searching for web crawler. This is also different from the algorithm for searching through multiple hyperlinks. But, practice getting the code?! I think it would be good to just think about it that way.
It can be displayed using python in VS Code. Before that, open a prompt window for the module and pip install bs4 requests
Download 2 modules.
beautifulSoup is a Python package for parsing HTML and XML. Parsing refers to the process of loading sentences into interpretable units, processing the necessary data on the web, and loading them to the server. requests is an HTTP library for the python programming language. It is also shown in the Wikipedia example, but when loading the page code, you can get it like reqeusts.get("url address").text.
+ Pylance report missing module source
If the Pylance extension recognizes a different version of Python in VS Code, the message “missing module” appears and import fails. This can be resolved by entering python using the Ctrl+Shift+P shortcut keys and selecting the python version that should be applied. In some cases, where re-launching was not possible, re-launching VS Code was sufficient.
+ If the terminal window shows that there is no module
There were cases where the terminal window showed that there were no installed modules. In this case, there are cases where the installation in the corresponding path is not reflected well.
(1) Uninstall and reinstall again
(2) This was resolved by installing from the location where pip.exe is located. The location of pip.exe is in the sub-Scripts folder in the path where python is installed.
4. Create a Telegram bot
Learning how to create a Telegram bot is an essential part of crawling. You can search for BotFather on Telegram and follow the manual to set up bot creation with /newbot. Done! Congratulations on your new bot. When the phrase appears, the token value appears.
After installing with pip install python-telegram-bot,
import telegram
(class) Bot(token: str , base_url: str | None = None, base_file_url: str | None = None, request: Request | None = None, private_key: bytes | None = None, private_key_password: bytes | None = None, defaults: Defaults | None = None)
Now you can use the property called Bot, and just enter the token value in parentheses as the token value you received after setting it earlier.
You can send content to the Telegram bot channel using sendMessage() with the above value. In the parentheses, just write the channel id value and the content you want to transmit. The channel id value can be obtained indirectly using getUpdates.
웹 크롤링 어디까지 허용되는가? (Web Crawling, Web Scrapping Issue)
No Module Named bs4 beautifulsoup python 파이썬 :: studyhard24
20초도 안걸리는 파이썬 설치 경로 알아내기 !! | 파이썬 설치 경로 확인
Visual Studio Code 에서 Python 패키지의 Unresolved Import Error 를 처리하는 방법
OKKY - 파이썬 pip 및 텔레그램 모듈 설치관련 질문입니다.
Comments
No comments yet. Be the first!