Competitor Price Monitoring and Tracking Software

Scrapy is the most commonly used library in Python for collecting online e-commerce price data. We will explain step by step how to develop a basic crawler using Scrapy to build your own price tracking software.

• How to Install Scrapy

First, ensure that the latest version of Python is installed on your computer. Then, install the Scrapy library for your project using the following command. You can run this command in the Python terminal or command prompt to complete the installation:

pip install scrapy

• Creating a Scrapy Project

To start using Scrapy, you first need to create a project. You can initiate a new Scrapy project with the following command:

scrapy startproject fiyat_takibi

This command creates a folder named “fiyat_takibi” and includes the basic Scrapy project files within it.

• Creating a Scrapy Spider

Now, we need to create a spider to extract product prices. In Scrapy, a spider is a class written to retrieve data from a specified web page.

Go to the fiyat_takibi/spiders directory within the project folder.

Create a new Python file, for example;

fiyat_spider.py.

Here is an example of a Scrapy crawler:

import scrapy

class FiyatSpider(scrapy.Spider):

    name = “fiyat_spider”

    # Define the URLs of your competitor’s product pages here;

    start_urls = [

        ‘https://www.ornek-site.com/urunler/urun-1’,

        ‘https://www.ornek-site.com/urunler/urun-2’,

        ‘https://www.ornek-site.com/urunler/urun-3’,

    ]

    def parse(self, response):

        # We are extracting the page title and product price;

        urun_adi = response.xpath(‘//h1[@class=”product-title”]/text()’).get()

        urun_fiyati = response.xpath(‘//span[@class=”price”]/text()’).get()

        # We are returning the data as output.

        yield {

            ‘urun_adi’: urun_adi,

            ‘urun_fiyati’: urun_fiyati.strip() if urun_fiyati else ‘Fiyat bulunamadı’,

        }

Explanation:

The code snippet above is written for a sample e-commerce site. You should run it separately for each competitor’s website by assigning unique names.

start_urls: Each competitor’s website links should be stored in separate files. Here, you define the URL lists of the product pages you want to track for competitor price monitoring.

parse: Scrapy runs the parse function for each URL to extract data from the page. Here, we use XPath expressions to retrieve the product name and price information from the page.

xpath: It is a method used to locate elements on a page. XPath provides access to XML and HTML structures.

yield: This tells Scrapy to return the data as output. The extracted data is usually saved in a JSON or CSV file.

• Running the Spider 

To run the spider, use the following command in the terminal or command prompt:

scrapy crawl fiyat_spider -o fiyatlar.json

This command runs the spider and saves the extracted data into the fiyatlar.json file. This file will contain product names and prices.

• Additional Features for Automated Price Tracking with Scrapy

1. Scheduler: You can use Scrapy with a cron job (Linux/Mac) or Windows Task Scheduler to automatically fetch prices at a specific time of the day.
2. Data Storage: Extracted data can be saved in CSV, JSON, or a database (such as MySQL, PostgreSQL, etc.). We recommend storing files in JSON format, as it offers both speed and space efficiency.
   At later stages of your project, running the system on a PostgreSQL database will allow you to generate advanced reports..