Arbitrage Betting Scrapers on Puppeteer
We had been long wondering about the sustainability of arbitrage betting. If this was a thing and how this could be automated and scaled. Thats why we gave it a go! For those who don’t yet know what arbitrage is, it is the way to place bets and win no matter the outcome. That is why it is also called sure betting.
We built an automated scraping system that crawls multiple online betting sites. It then gathers the data to calculate possible arbitrage. The end goal was to asynchronously notify the user to take action.
Single website scraper
First we built a web scraper with Puppeteer. With a few lines of code you can have it visiting a page and interacting with it in all ways that a real user could. The majority of the work required is coming up with the right CSS selectors for the elements you ll need to read or interact with.
Automating for multiple websites
After getting this working for one site, we made it working for multiple. We located the code referring to this site’s specifics (e.g. CSS selectors, urls, clicking on “accept cookies” etc) and we refactored it to be dynamically created from a config file. Now all we had to do to get betting data from other websites was to create just the configs for them! Creating the configs was a matter of visiting the page and finding out the selectors to use.
Up to now we have a puppeteer application that spawns multiple browsers to fetch betting data from multiple websites. The next step is to calculate the arbitrages, if any. We persist the data in a database and created a script to process those. There’s a lot trickyness to this but in the end we made it working and found our first arbitrage!
- Automate account management to all betting sites
- Automate bet placements coordinated from the arbitrage algorithm
- UI to enter the maximum and minimum limits for betting as well as betting frequency
- UI to add/remove websites without coding
- UI to add/remove more sports per website