Wikipedia offers AI developers a training dataset to maybe get scraper bots off its back
Wikipedia has been struggling with the impact that AI crawlers — bots that are scraping text and multimedia from the encyclopedia to train generative artificial intelligence models — have been having on its servers, leading to increased costs and slower load times for human users in some cases. Perhaps in an effort to stop the bots from pummeling the public Wikipedia website and soaking up too much bandwidth, the Wikimedia Foundation (which manages Wikipedia's data) is offering AI developers a dataset they can freely use.
The organization has teamed up with Kaggle, a data science platform, to offer up a beta release of a structured dataset in both English and French. According to Google — which owns Kaggle — the dataset is formatted for machine learning to make it more useful for training, development and data science.
Wikimedia Enterprise notes that the dataset includes "abstracts, short descriptions, infobox-style key-value data, image links and clearly segmented article sections." There are no references or other "non-prose elements," such as video clips. The lack of references could make the issue of attribution for information in the dataset somewhat foggy. However, Wikimedia Enterprise (a part of the Wikimedia Foundation that seeks to make Wikipedia data available through APIs) says that the content in the dataset is freely licensed under Creative Commons, the public domain and so on since it's all from Wikipedia.This article originally appeared on Engadget at https://www.engadget.com/ai/wikipedia-offers-ai-developers-a-training-dataset-to-maybe-get-scraper-bots-off-its-back-143255593.html?src=rss
Rating
Innovation
Pricing
Technology
Usability
We have discovered similar tools to what you are looking for. Check out our suggestions for similar AI tools.
Wikipedia cancels plan to test AI summaries after editors skewer the idea
<p>Wikipedia is backing off a plan to test AI article summaries. Earlier this month, the platform announced plans to trial the feature for about 10 percent of mobile web visitors. To say they we [...]
Best Memorial Day tech sales from Amazon, Apple, Samsung and others that yo
<p>It might surprise you, but Memorial Day can be a great time to shop for tech. In years past, we've seen solid Memorial Day sales on all manner of gadgets including many of our favorite tablet [...]
Best Memorial Day tech sales from Amazon, Apple, Dyson and others that you
<p>It might surprise you, but Memorial Day can be a great time to shop for tech. In years past, we've seen solid Memorial Day deals on all manner of gadgets including many of our favorite tablet [...]
Best Memorial Day tech sales from Amazon, Apple, Samsung, Dyson and others
<p>It might surprise you, but Memorial Day can be a great time to shop for tech. In years past, we've seen solid Memorial Day sales on all manner of gadgets including many of our favorite tablet [...]
The best Memorial Day tech sales from Amazon, Apple, Samsung, Dyson and oth
<p>It might surprise you, but Memorial Day can be a great time to shop for tech. In years past, we've seen solid Memorial Day sales on all manner of gadgets including many of our favorite tablet [...]