Web Scraping is the process of extracting content from human-readable websites in order to import it into local storage such as databases or CSV Files. The process of data extraction and its design is time-consuming requiring an analysis of the website, data representation of the objects comprising its structure (DOM), HTML tags, and the Cascading Style Sheets (CSS) classes. To support this process we aim at providing automation. In this paper, we propose a pattern mining technique to scrap news and blog websites by recognizing title and body based on a content structure pattern. This approach consists of three steps, i.e.: extracting news website structure, constructing a pattern of HTML content, and implementing the pattern as a set of rules in web scraping. Our approach is a simple, general, and straightforward way to extract articles that consist of the title, the body of any blogs, or news websites.

This paper presents a novel solution aimed at enhancing news web pages for seamless integration into the Semantic Web. By utilizing advanced pattern mining techniques alongside OpenAI’s GPT-3, we rewrite news articles to improve their readability and accessibility for Google News aggregators. Our approach is characterized by its methodological rigour and is evaluated through quantitative metrics, validated using Google’s Rich Results Test API to confirm adherence to Google’s structured data guidelines. In this process, a “Pass” in the Rich Results Test is taken as an indication of eligibility for rich results, demonstrating the effectiveness of our generated structured data. The impact of our work is threefold: it advances the technological integration of a substantial segment of the web into the Semantic Web, promotes the adoption of Semantic Web technologies within the news sector, and significantly enhances the discoverability of news articles in aggregator platforms. Furthermore, our solution facilitates the broader dissemination of news content to diverse audiences. This submission introduces an innovative solution substantiated by empirical evidence of its impact and methodological soundness, thereby making a significant contribution to the field of Semantic Web research, particularly in the context of news and media articles.

Telegram chatbots are one of the main sections in Telegram applications that get more attention and provide huge business opportunities for companies nowadays. Also, Machine learning algorithms have been used in numerous fields and industries and one of the main growing markets is real estate. It is already helping real estate agents to respond more quickly to clients’ questions and develop landing pages that fit customers’ needs. However, one of the main time-consuming stages for a real estate agent is the process of answering questions about price per location and knowing which locations clients are interested in. In order to solve these problems, we have built a telegram chatbot to answer customers of any real estate agent with the price of a real estate property based on their current Geolocation. This bot can predict the price of a real estate property using room numbers, Geolocation, and surface area in square meters. These are the inputs for machine learning algorithms to give an approximate price. The bot can predict based on the source of the data-set, for example, we have collected our data from classified ads website for real estate in Amman, Jordan. We have used python and web scraping library for data extraction, cleaning, and transformation. As a result, by using Geolocation we have increased the model accuracy to 1.3X and our bot can be replicated in any markets based on the dataset. We believe our work can be taken to different markets and make real estate agent’s job easy and more profitable by changing leads to potential customers thought answering their questions.

Irrelevant elements like ads, menus, and footers in web pages hinder data extraction and reduce the performance of Retrieval-Augmented Generation (RAG) systems in Large Language Models (LLMs). This paper tackles the challenge of accurately identifying and extracting the main content from web pages to enhance the efficiency of these systems. We present a novel mathematical model and algorithm that leverages the Document Object Model (DOM) structure, effectively isolating relevant content with high accuracy. Our approach is language-neutral and performs well across diverse languages, including those with complex tokenization, such as Arabic. To validate the model, we created a dataset from 500 websites, allowing for comprehensive evaluation and benchmarking. The algorithm’s practical application demonstrates a reduction in token usage for LLM tasks, contributing to cost-effectiveness. This work introduces a robust, open-source tool for the academic and commercial communities, fostering further innovation in web content extraction and information retrieval.