/u/avatar/76A0On-R5D7X01){kind=small}
记录一些常见的爬虫 UA，方便在爬虫访问是进行统计
| 主体 | user-agent | 用途 |
|---|---|---|
| googlebot | 搜索引擎 | |
| google-structured-data-testing-tool | 测试工具 | |
| Mediapartners-Google | Adsense广告网页被访问后，爬虫就来访 | |
| Microsoft | bingbot | 搜索引擎 |
| Linked | linkedinbot | 应用内搜索 |
| 百度 | baiduspider | 搜索引擎 |
| 奇虎 360 | 360Spider | 搜索引擎 |
| 搜狗 | Sogou Spider | 搜索引擎 |
| Yahoo | Yahoo! Slurp China | 搜索引擎 |
| Yahoo | Yahoo! Slurp | 搜索引擎 |
| 头条 | Bytespider | 搜索引擎 |
| twitterbot | 应用内搜索 | |
| facebookexternalhit | 应用内搜索 | |
| - | rogerbot | - |
| - | embedly | - |
| Quora | quora link preview | - |
| - | showyoubot | - |
| - | outbrain | - |
| - | - | |
| - | slackbot | - |
| - | vkShare | - |
| - | W3C_Validator | - |
nginx 判断：
if ($http_user_agent ~* "googlebot|bingbot|yandex|baiduspider|360Spider|Sogou Spider|Bytespider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest\/0\.|pinterestbot|slackbot|vkShare|W3C_Validator|whatsapp") {
## do something
}
/u/avatar/aopRLw-L62b3A){kind=small}
/users/-/avatars/eef1bb43-c8f8-4ef4-9dfb-e5790818e001){kind=small}
/u/avatar/Mp8Gxp-JKmz8WO){kind=small}
/users/-/avatars/1a95850c-9fd0-4751-86a2-f7c8cf9c988c){kind=small}
/u/avatar/KKoP5V-OJ0gYAN){kind=small}
/users/-/avatars/258f9a50-cd84-48f9-96e1-f75e0ce3719b){kind=small}
/users/-/avatars/4cb682f9-1c9d-4782-ae93-6eb7ff531054){kind=small}