大多数已经说过了，但是你考虑过CloudFlare的保护吗?我的意思是:

其他公司可能也这么做，CloudFlare是我知道的唯一一家。

我很确定这会使他们的工作复杂化。我还曾经因为速率限制而试图废弃一个受CloudFlare保护的网站的数据(我使用了简单的AJAX请求循环)，导致IP被自动禁用了4个月。

你可以做一些事情来防止屏幕抓取。有些不是很有效，而另一些(验证码)是，但阻碍可用性。你必须记住，它也可能阻碍合法的网站刮刀，如搜索引擎索引。

然而，我认为如果你不希望它被删除，这意味着你也不希望搜索引擎索引它。

这里有一些你可以尝试的方法:

Show the text in an image. This is quite reliable, and is less of a pain on the user than a CAPTCHA, but means they won't be able to cut and paste and it won't scale prettily or be accessible. Use a CAPTCHA and require it to be completed before returning the page. This is a reliable method, but also the biggest pain to impose on a user. Require the user to sign up for an account before viewing the pages, and confirm their email address. This will be pretty effective, but not totally - a screen-scraper might set up an account and might cleverly program their script to log in for them. If the client's user-agent string is empty, block access. A site-scraping script will often be lazily programmed and won't set a user-agent string, whereas all web browsers will. You can set up a black list of known screen scraper user-agent strings as you discover them. Again, this will only help the lazily-coded ones; a programmer who knows what he's doing can set a user-agent string to impersonate a web browser. Change the URL path often. When you change it, make sure the old one keeps working, but only for as long as one user is likely to have their browser open. Make it hard to predict what the new URL path will be. This will make it difficult for scripts to grab it if their URL is hard-coded. It'd be best to do this with some kind of script.

如果我必须这样做，我可能会结合使用后三种方法，因为它们最大限度地减少了对合法用户的不便。然而，你必须接受这样的事实:你不可能用这种方式屏蔽所有人，一旦有人想出了绕过它的方法，他们就可以永远地刮掉它。我猜你可以在发现他们的时候屏蔽他们的IP地址。

我已经做了很多网络抓取，并在我的博客上总结了一些技巧来阻止网络抓取，基于我觉得讨厌的东西。

这是你的用户和刮刀者之间的权衡。如果你限制IP，使用验证码，要求登录，等等，你会给刮刀器带来困难。但这也可能会赶走真正的用户。

你真的没有什么办法可以完全防止这种情况。刮刮者可以伪造他们的用户代理，使用多个IP地址等，并以正常用户的身份出现。你唯一能做的就是让文本在页面加载时不可用——用图像、flash或JavaScript加载。然而，前两个都是糟糕的想法，如果JavaScript没有为一些常规用户启用，最后一个将是可访问性问题。

如果他们对你的网站进行了猛烈的攻击，并且浏览了你所有的页面，你可以做一些速率限制。

不过还是有希望的。刮刮器依赖于您的网站的数据是一致的格式。如果你能随机分配，它可能会打碎他们的刮刀。比如在每次加载时更改页面元素的ID或类名等。但这需要做很多工作，我不确定是否值得。即便如此，只要有足够的投入，他们可能就能解决这个问题。

快速解决这个问题的方法是设置一个陷阱。

Make a page that if it's opened a certain amount of times or even opened at all, will collect certain information like the IP and whatnot (you can also consider irregularities or patterns but this page shouldn't have to be opened at all). Make a link to this in your page that is hidden with CSS display:none; or left:-9999px; positon:absolute; try to place it in places that are less unlikely to be ignored like where your content falls under and not your footer as sometimes bots can choose to forget about certain parts of a page. In your robots.txt file set a whole bunch of disallow rules to pages you don't want friendly bots (LOL, like they have happy faces!) to gather information on and set this page as one of them. Now, If a friendly bot comes through it should ignore that page. Right but that still isn't good enough. Make a couple more of these pages or somehow re-route a page to accept differnt names. and then place more disallow rules to these trap pages in your robots.txt file alongside pages you want ignored. Collect the IP of these bots or anyone that enters into these pages, don't ban them but make a function to display noodled text in your content like random numbers, copyright notices, specific text strings, display scary pictures, basically anything to hinder your good content. You can also set links that point to a page which will take forever to load ie. in php you can use the sleep() function. This will fight the crawler back if it has some sort of detection to bypass pages that take way too long to load as some well written bots are set to process X amount of links at a time. If you have made specific text strings/sentences why not go to your favorite search engine and search for them, it might show you where your content is ending up.

无论如何，如果你从战术和创造性的角度思考，这可能是一个很好的起点。最好的办法就是学习机器人是如何工作的。

我还会考虑打乱一些ID或页面元素上的属性显示方式:

<a class="someclass" href="../xyz/abc" rel="nofollow" title="sometitle"> 

每次都会改变它的形式，因为一些机器人可能会在你的页面或目标元素中寻找特定的模式。

<a title="sometitle" href="../xyz/abc" rel="nofollow" class="someclass"> 

id="p-12802" > id="p-00392"

我同意上面的大部分帖子，我想补充的是，你的网站对搜索引擎越友好，它就越容易被抓取。你可以尝试做一些让刮刀器更难的事情，但它也可能影响你的搜索能力……当然，这取决于你想让你的网站在搜索引擎上排名有多好。