While many organisations are utilising ‘web scraping’ to drive business decisions, misplaced legal concerns are discouraging others from leveraging the data gathering technique, according to research from Oxylabs and Censuswide.
The research surveyed over 1000 UK and US-based senior decision makers, including CTOs, VPs of Engineering, Chief Data Officers, Heads of Business Intelligence, and Heads of Data & Analytics.
The main concern of businesses that have not implemented web scraping has been potential legal complications. More than half (53.06%) of all organisations that haven’t implemented web scraping cite legal risks as the main reason.
Lack of technical expertise is the second most cited reason (39.59%), followed closely by budget restrictions (30.20%). Web scraping has become more accessible than ever, as only 1 in 5 businesses cites complicated technology as an excuse not to implement it (20.82%).
According to Oxylabs’ Head of Legal, Denas Grybauskas, web scraping is still misunderstood and retailers should be aware of the huge benefits it offers – especially during the golden quarter.
He said: “It has been very tricky to overcome the perception that behind every data point lies a legal risk, particularly when it comes to web scraping. Even though organisations should always consult with legal professionals, there are best practices within the industry that, when followed, minimise most risks associated with data collection. These results indicate a need to increase our efforts and educate the industry about the legal aspects of web scraping.
“To help educate business leaders, organisations can turn to initiatives such as Ethical Web Data Collection Initiative (EWDCI), which aims to reach out to businesses and educate them on the importance of ethical data collection, as well as educate users and legislators worldwide in the interest of informed, empowered action. This will encourage industry-led stewardship of critical and widely used technology in order to address current challenges and position the industry for future success.”
Grybaukas added: “Retailers can use web scraping to improve customer experience, which is extremely important during the golden quarter. Companies can maximise profit while staying competitive and without losing the customer. This can be achieved by scraping product and market information to fine-tune the strategies to increase customer satisfaction through areas such as recommendation engines. Since this data also enables you to keep a thorough check on the not-so-static market at large, it helps you create a dynamic pricing strategy to keep up with the changes and help maximise the profit.”
Lack of technical expertise is a greater barrier to US-based companies (46.77% vs 37.16%), while budget constraints affect the UK more (32.24% vs 24.19%). Risks of legal complications is the greatest factor for both regions with nearly no differences (>50% for both).
Grybauskas said: “The web scraping industry still suffers from several misconceptions, and organisations need to be educated better on this issue. Web scraping is a powerful tool for analysing websites, and its benefits far outweigh the drawbacks. Retailers can gain insights into how to increase traffic and sales, manifest ideal conversation rates, and, ultimately, save their businesses and give them a new and refined appearance by analysing public web data”.
To download your copy of the Oxylabs’ Alternative Data Defines Competition in the UK & US eCommerce research findings, please visit here.
