New research finds that AI is having a positive impact on the data collection industry in the areas of target unblocking and parsing.

VILNIUS, LITHUANIA / ACCESSWIRE / November 20, 2024 / 86% of data collection professionals believe AI adoption in web scraping saves time and resources as the demand for web intelligence continues to rise. This is one of the key findings from the new Oxylabs white paper, Meeting the Increasing Business Demand for Web Intelligence , which demonstrates the growth and impact of web intelligence solutions.

In cooperation with Censuswide, Oxylabs surveyed 506 software engineers and web scraping professionals from UK and US companies, gathering insights on the issues faced when collecting web intelligence and the impact it brings to businesses.

77% of respondents stated that they had tried using AI-powered web scraping solutions, with those in the US more likely to use AI for web scraping than those in the UK (56% vs. 38%). The discrepancy might partly be explained by another finding of the survey. The share of professionals who fix parsers every day as a result of changing website structures was found to be greater in the US at 38% over 24% in the UK. The more scraping professionals face such an intense parsing schedule, the more likely they are to turn to AI for assistance.

The web intelligence community has welcomed AI innovations and is prepared for the change these innovations can bring. Only 4% of the respondents stated that AI did not meet their expectations. Meanwhile, 86% of those who tried AI solutions for web scraping confirmed that it helped them save time and resources. Almost half of scraping professionals (47%) use AI-powered solutions regularly, with 77% having tried using AI solutions and another 20% claiming that they are keen to try them, leaving only 3% of web data collection professionals who have no interest in utilizing AI.

Juras Juršėnas, chief operations officer at Oxylabs, said, "In many industries, the true extent of AI's potential impact and its ability to provide a return on the huge investment it had already accumulated is still up for debate. However, large-scale web data collection seems to be perfectly suited for AI enhancements. Building and maintaining data pipelines consists of multiple operations and repetitive tasks that AI solutions can automate or improve. While we are still at an early stage of AI adoption, it is both exciting and vital for web scraping companies to closely watch the emerging AI/ML innovations and trends of leveraging these tools."

When asked about the web scraping tasks respondents used AI to complete, 67% identified building parsers, while 59% used AI for target unblocking and 52% for building crawling logic. This suggests that AI adopters in web scraping are testing the hypothesis that AI can streamline many if not most, tech tasks. 43% of scraping professionals stated AI is best suited for target unblocking, suggesting that AI is perceived as most applicable in dealing with sophisticated tasks.

" Web unblocking is one of the first areas where we have utilized AI technologies," added Aleksandras Šulženko, product owner at Oxylabs. "However, while web unblocking becomes a challenge in specific, although important, cases, parsing is a crucial part of every scraping pipeline. Thus, an AI solution for building parsers is even more widely applicable and brings constant value to the customer. "

The feedback regarding AI usage from web intelligence gathering professionals has been overwhelmingly positive, but its application in parsing has not produced revolutionary results that would add exceptional value. Parsing has remained time-consuming and challenging even when AI is applied, indicating that there was no suitable AI-based solution for parsing in the market when this survey was conducted. The introduction of Oxylabs' AI-powered OxyCopilot has filled this gap, allowing data parsers to be built in minutes and helping create initial scraping requests after providing only minimal parameters.

Julius Černiauskas, chief executive officer at Oxylabs, concluded, "The demand for high-quality public web data is accelerating, and with the advancements in AI model development and increased usage of tools based on these models, this trend will continue well into the future."

To learn more about this research's results, download the free white paper from Oxylabs' official website.

-ENDS-

About Oxylabs

Established in 2015, Oxylabs is a leading web intelligence platform and premium proxy provider, enabling companies of all sizes to utilize the power of big data. Constant innovation, an extensive patent portfolio, and a focus on ethics have allowed Oxylabs to become a global leader in the web intelligence collection industry and forge close ties with dozens of Fortune Global 500 companies. In 2022, 2023 and 2024, Oxylabs was named Europe's fastest-growing web intelligence acquisition company in the Financial Times' FT 1000 list. For more information, please visit: https://oxylabs.io/

Media Contacts

Benjamin Hart / Sasha Arion
Spreckley
Tel: (0)207 388 9988
Email:  oxylabs@spreckley.co.uk

Vytautas Kirjazovas 
Oxylabs.io
Tel: +370 655 34419
Email: press@oxylabs.io

SOURCE: Oxylabs