I'm not familiar with the mechanics of web crawlers, so hopefully somebody can shed some light.
Would there be value in a search engines for specific topics? For example, movie reviews and blogs. Assuming you had a way to determine if a page is a review or blog, would it make indexing easier to ignore the rest? Could a tailored search engine be tweaked to give better results?
I kind of think it would be useful, because some websites I use have really great search tools (e.g. ones with tags). But those are only for the website itself, so there's no crawling.
Would there be value in a search engines for specific topics? For example, movie reviews and blogs. Assuming you had a way to determine if a page is a review or blog, would it make indexing easier to ignore the rest? Could a tailored search engine be tweaked to give better results?
I kind of think it would be useful, because some websites I use have really great search tools (e.g. ones with tags). But those are only for the website itself, so there's no crawling.