Need help scraping a MediaWiki site


I’m trying to scrape a MediaWiki site that we use for a support wiki, but the script is having difficulties following links/finding pages. I used this example to build my code - OpenAI API

Thank you for any feedback!

You may have to pay, but please look at the Langchain APIFY module. It is integrated with APIFY who does the crawl for you

1 Like

Definitely look at

I struggled with a couple of scrapers for several days, and didn’t even try Apify because of the javascript requirements. uses AI to set up scraper and, relatively speaking, couldn’t be easier to use. It also has integrations including Google Sheets and Zapier.