记录一些常见的爬虫 UA，方便在爬虫访问是进行统计

| 主体        | user-agent                          | 用途                    |
| :-------- | :---------------------------------- | :-------------------- |
| Google    | googlebot                           | 搜索引擎                  |
| Google    | google-structured-data-testing-tool | 测试工具                  |
| Google    | Mediapartners-Google                | Adsense广告网页被访问后，爬虫就来访 |
| Microsoft | bingbot                             | 搜索引擎                  |
| Linked    | linkedinbot                         | 应用内搜索                 |
| 百度        | baiduspider                         | 搜索引擎                  |
| 奇虎 360    | 360Spider                           | 搜索引擎                  |
| 搜狗        | Sogou Spider                        | 搜索引擎                  |
| Yahoo     | Yahoo! Slurp China                  | 搜索引擎                  |
| Yahoo     | Yahoo! Slurp                        | 搜索引擎                  |
| 头条        | Bytespider                          | 搜索引擎                  |
| Twitter   | twitterbot                          | 应用内搜索                 |
| Facebook  | facebookexternalhit                 | 应用内搜索                 |
| -         | rogerbot                            | -                     |
| -         | embedly                             | -                     |
| Quora     | quora link preview                  | -                     |
| -         | showyoubot                          | -                     |
| -         | outbrain                            | -                     |
| -         | pinterest                           | -                     |
| -         | slackbot                            | -                     |
| -         | vkShare                             | -                     |
| -         | W3C_Validator                       | -                     |

nginx 判断：

```conf
if ($http_user_agent ~* "googlebot|bingbot|yandex|baiduspider|360Spider|Sogou Spider|Bytespider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest\/0\.|pinterestbot|slackbot|vkShare|W3C_Validator|whatsapp") {
## do something
}
```

常见爬虫 UserAgent 汇总

单独判断 UA 的话没法确定是否为真实的 “某某爬虫”，严格的话要对客户端 IP 做 DNS 分析确定，有很多爬虫伪造 googlebot 去爬别人的内容，比如 [fake-useragent](https://pypi.org/project/fake-useragent/) 之类

标识符

483DAB9E-B587-4DAE-936C-201F5BC4BCBA

备注

Error Domain=kCFStreamErrorDomainSSL Code=-9806"(null)" UserInfo=/NSLocalizedRecoverySuggestion=Errorcode definition can be found in Apple's SecureTransport.h}

请求信息

CONNECT m.m******(手动打码).pt:443 HTTP/1.1

请求头部

Host: m.m*******.ptConnection: keep-alive

发送请求头部

未发送

服务端开始响应

未响应

请求完成2024/04/26 10:11:40.529

 app无法登录，抓包数据。网页版可以登录，求分析求解

前些天一直在找便宜的云服务器，偶然间发现一免费的云平台，这里推荐给大家。

就是阿贝云，网站是https://www.abeiyun.com

说一下感受

1、阿贝云虽然是一小众的云平台，重在免费，该有的东西都有，还能免备案这是最舒服的。

2、使用了一天，上了项目，网速不慢。

3、唯一要指出来就是磁盘太小。

分享免费云服务器

拉取了rustbuster的v3.0.3版本文件, 二进制文件无法使用, 执行结果如下:

```bash
[root_cn@archlinux target]$ ./rustbuster-v3.0.3-x86_64-unknown-linux-gnu
./rustbuster-v3.0.3-x86_64-unknown-linux-gnu: error while loading shared libraries: libssl.so.1.1: cannot open shared object file: No such file or directory
```

拉取了main源码, 想自行尝试编译, 然后卡在了openssl, 以下为报错信息:

```rust
The following warnings were emitted during compilation:

warning: openssl-sys@0.9.47: build/expando.c:4:24: error: pasting "RUST_VERSION_OPENSSL_" and "(" does not give a valid preprocessing token
warning: openssl-sys@0.9.47:     4 | #define VERSION2(n, v) RUST_VERSION_##n##_##v
warning: openssl-sys@0.9.47:       |                        ^~~~~~~~~~~~~
warning: openssl-sys@0.9.47: build/expando.c:5:23: note: in expansion of macro ‘VERSION2’
warning: openssl-sys@0.9.47:     5 | #define VERSION(n, v) VERSION2(n, v)
warning: openssl-sys@0.9.47:       |                       ^~~~~~~~
warning: openssl-sys@0.9.47: build/expando.c:10:1: note: in expansion of macro ‘VERSION’
warning: openssl-sys@0.9.47:    10 | VERSION(OPENSSL, OPENSSL_VERSION_NUMBER)
warning: openssl-sys@0.9.47:       | ^~~~~~~

error: failed to run custom build command for `openssl-sys v0.9.47`

Caused by:
  process didn't exit successfully: `/mnt/d/work/file/rust/rustbuster/target/debug/build/openssl-sys-c5a482a9e0e49574/build-script-main` (exit status: 101)
  --- stdout
  cargo:rustc-cfg=const_fn
  cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_GNU_OPENSSL_LIB_DIR
  X86_64_UNKNOWN_LINUX_GNU_OPENSSL_LIB_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_LIB_DIR
  OPENSSL_LIB_DIR unset
  cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_GNU_OPENSSL_INCLUDE_DIR
  X86_64_UNKNOWN_LINUX_GNU_OPENSSL_INCLUDE_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_INCLUDE_DIR
  OPENSSL_INCLUDE_DIR unset
  cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_GNU_OPENSSL_DIR
  X86_64_UNKNOWN_LINUX_GNU_OPENSSL_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_DIR
  OPENSSL_DIR unset
  cargo:rustc-link-lib=ssl
  cargo:rustc-link-lib=crypto
  OPT_LEVEL = Some("0")
  TARGET = Some("x86_64-unknown-linux-gnu")
  HOST = Some("x86_64-unknown-linux-gnu")
  CC_x86_64-unknown-linux-gnu = None
  CC_x86_64_unknown_linux_gnu = None
  HOST_CC = None
  CC = None
  CFLAGS_x86_64-unknown-linux-gnu = None
  CFLAGS_x86_64_unknown_linux_gnu = None
  HOST_CFLAGS = None
  CFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("true")
  CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
  running: "cc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-E" "build/expando.c"
  cargo:warning=build/expando.c:4:24: error: pasting "RUST_VERSION_OPENSSL_" and "(" does not give a valid preprocessing token
  cargo:warning=    4 | #define VERSION2(n, v) RUST_VERSION_##n##_##v
  cargo:warning=      |                        ^~~~~~~~~~~~~
  cargo:warning=build/expando.c:5:23: note: in expansion of macro ‘VERSION2’
  cargo:warning=    5 | #define VERSION(n, v) VERSION2(n, v)
  cargo:warning=      |                       ^~~~~~~~
  cargo:warning=build/expando.c:10:1: note: in expansion of macro ‘VERSION’
  cargo:warning=   10 | VERSION(OPENSSL, OPENSSL_VERSION_NUMBER)
  cargo:warning=      | ^~~~~~~
  exit status: 1

  --- stderr
  thread 'main' panicked at /home/root_cn/.cargo/registry/src/mirrors.sjtug.sjtu.edu.cn-4f7dbcce21e258a2/openssl-sys-0.9.47/build/main.rs:126:13:

  Header expansion error:
  Error { kind: ToolExecError, message: "Command \"cc\" \"-O0\" \"-ffunction-sections\" \"-fdata-sections\" \"-fPIC\" \"-g\" \"-fno-omit-frame-pointer\" \"-m64\" \"-Wall\" \"-Wextra\" \"-E\" \"build/expando.c\" with args \"cc\" did not execute successfully (status code exit status: 1)." }

  Failed to find OpenSSL development headers.

  You can try fixing this setting the `OPENSSL_DIR` environment variable
  pointing to your OpenSSL installation or installing OpenSSL headers package
  specific to your distribution:

      # On Ubuntu
      sudo apt-get install libssl-dev
      # On Arch Linux
      sudo pacman -S openssl
      # On Fedora
      sudo dnf install openssl-devel

  See rust-openssl README for more information:

      https://github.com/sfackler/rust-openssl#linux

  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
```

经过检查openssl与rust均为最新版本, 

```bash
[root_cn@archlinux target]$ sudo pacman -S rust
[sudo] password for root_cn:
warning: rust-1:1.77.2-1 is up to date -- reinstalling
resolving dependencies...
looking for conflicting packages...

Package (1)  Old Version  New Version  Net Change

extra/rust   1:1.77.2-1   1:1.77.2-1     0.00 MiB

Total Installed Size:  249.41 MiB
Net Upgrade Size:        0.00 MiB

:: Proceed with installation? [Y/n] n
[root_cn@archlinux target]$ sudo pacman -S openssl
warning: openssl-3.2.1-1 is up to date -- reinstalling
resolving dependencies...
looking for conflicting packages...

Package (1)   Old Version  New Version  Net Change

core/openssl  3.2.1-1      3.2.1-1        0.00 MiB

Total Installed Size:  11.45 MiB
Net Upgrade Size:       0.00 MiB

:: Proceed with installation? [Y/n] n
```

在网上找了一下, 查到资料, 在[rust-openssl的issues](https://github.com/sfackler/rust-openssl/issues/1227)中提到类似报错, 得到解释如下:

> If you are cross compiling to MUSL, you either need to point openssl-sys to the location that you have a copy of OpenSSL cross compiled to MUSL, or tell it to build a copy of OpenSSL itself with the vendored feature.
>
>	如果你是交叉编译到MUSL，你需要将openssl-sys指向你有一个交叉编译到MUSL的OpenSSL副本的位置，或者告诉它用vendored特性构建一个OpenSSL本身的副本。

由于不会rust, 依靠关键词"交叉编译"+"cargo"搜索到了[cargo交叉编译rust程序](https://blog.csdn.net/tyc880616/article/details/104371502), 顺着找到文件:

```
[root_cn@archlinux .cargo]$ cat ~/.cargo/config
[source.crates-io]
replace-with = 'sjtu' # 指定使用下面哪个源，修改为source.后面的内容即可

# 中国科学技术大学
[source.ustc]
registry = "https://mirrors.ustc.edu.cn/crates.io-index"

# 上海交通大学
[source.sjtu]
registry = "https://mirrors.sjtug.sjtu.edu.cn/git/crates.io-index/"

# 清华大学
[source.tuna]
registry = "https://mirrors.tuna.tsinghua.edu.cn/git/crates.io-index.git"

# rustcc社区
[source.rustcc]
registry = "https://code.aliyun.com/rustcc/crates.io-index.git"
```

没有找到交叉编译的设置, 由于本人不会rust, 线索断了, 尝试向大家寻求一波帮助, 非常感谢阅读.



急切寻求rust编译问题帮助(archlinux, openssl)

永久免费，无需注册

[https://good.xjai.top](https://good.xjai.top)

分享免费GPT4站点

[MoienTajik/AspNetCore-Developer-Roadmap (github.com)](https://github.com/MoienTajik/AspNetCore-Developer-Roadmap/blob/master/ReadMe.zh-Hant.md)

![](https://github.com/MoienTajik/AspNetCore-Developer-Roadmap/raw/master/aspnetcore-developer-roadmap.zh-Hant.png)



【轉貼】ASP.NET Core Developer Roadmap in 2024

南京大学数字金融 - 新华报业集团联合诚邀全职博士后！

【研究方向涵盖】

数智传播、数智金融经济、数据金融经济、数智政务、数智监管治理，以及“大数据、大模型、大计算”全新范式等方向理论与实践

 【具体研究领域】

包括 Bigdata4SocialScience、AI4SocialScience、LLM4SocialScience、Blockchain4SocialScience，以及先进信息技术融合创新在数智金融经济、数智传播社会、数智健康医疗、数智政务治理中的应用。

【薪资待遇优渥】

年薪可达 30 万人民币左右，另有绩效奖金及江苏省和南京市人才补贴。具体薪资需与HR面议，总收入依据专业背景和技能经验而定（计算机 AI 背景者薪资更优）。

【欢迎学科交叉、专业背景不限】

不限制专业背景，致力于构建一个多元、开放、包容的科研环境，因此，欢迎计算机、软件、大数据、人工智能、区块链、隐私计算、数据安全、数据科学等背景的博士人才，也热切邀请具有人文社科背景的博士加入我们的团队，共同探索信息技术与社会科学的交叉前沿。也欢迎其他编程能力强的理工科博士毕业生（即使没有AI等背景也欢迎）。

【南京大学研究中心】

南京大学研究中心为双院士领军的校级科研和创新平台，详情请见：fintech.nju.edu.cn。

【团队氛围】

学科交叉、理论与实践相结合、国际化视野、行业前沿视角和项目。

【联系我们】

请感兴趣的朋友将个人简历发送至邮箱:blockchainfintech@126.com。

【更多招募】 南京大学研究中心招收硕士生、博士生、博士后、青年教师、研究员，也欢迎访问、客座、联培以及科研助理（有些岗位可兼职或远程合作）等

欢迎加入我们，您将在这个富有挑战性和机遇的环境中，开启广阔的职业发展道路，实现个人价值与学术梦想的双赢！

期待您的加入，携手共创数智未来！

南京大学研究中心诚募博士后



![](https://image.hackertalk.net/images/62537abc-405f-44da-a67e-9f18082f3d74)















sqlmap -u "xxxxx.com/#/login" --all  --delay 1 --batch --random-agent

[CRITICAL]can't establish SSL connection.

![](https://image.hackertalk.net/images/4a856cac-750c-4972-aa1f-4bf02f000394)

sqlmap -r win.txt --dbs

**

[CRITICAL] anti-CSRF token 'XSRF-TOKEN' can't be found 


**




主体	user-agent	用途
Google	googlebot	搜索引擎
Google	google-structured-data-testing-tool	测试工具
Google	Mediapartners-Google	Adsense广告网页被访问后，爬虫就来访
Microsoft	bingbot	搜索引擎
Linked	linkedinbot	应用内搜索
百度	baiduspider	搜索引擎
奇虎 360	360Spider	搜索引擎
搜狗	Sogou Spider	搜索引擎
Yahoo	Yahoo! Slurp China	搜索引擎
Yahoo	Yahoo! Slurp	搜索引擎
头条	Bytespider	搜索引擎
Twitter	twitterbot	应用内搜索
Facebook	facebookexternalhit	应用内搜索
-	rogerbot	-
-	embedly	-
Quora	quora link preview	-
-	showyoubot	-
-	outbrain	-
-	pinterest	-
-	slackbot	-
-	vkShare	-
-	W3C_Validator	-