Search is a much needed discovery tool on any website. Doesnt mean that it has to be costly. With search technology maturing, Expertrec offers a high quality search experience at an affordable cost.
Algolia - Algolia's Search API makes it easy to deliver a great search experience in your apps & websites. Algolia Search provides hosted full-text, numerical, faceted and geolocalized search.
Scrapy - Scrapy | A Fast and Powerful Scraping and Web Crawling Framework
ElasticSearch - Elasticsearch is an open source, distributed, RESTful search engine.
StormCrawler - StormCrawler is an open source SDK for building distributed web crawlers with Apache Storm.
Apache Solr - Solr is an open source enterprise search server based on Lucene search library, with XML/HTTP and...
Apache Nutch - Apache Nutch is a highly extensible and scalable open source web crawler software project.