记录一些常见的爬虫 UA，方便在爬虫访问是进行统计

| 主体        | user-agent                          | 用途                    |
| :-------- | :---------------------------------- | :-------------------- |
| Google    | googlebot                           | 搜索引擎                  |
| Google    | google-structured-data-testing-tool | 测试工具                  |
| Google    | Mediapartners-Google                | Adsense广告网页被访问后，爬虫就来访 |
| Microsoft | bingbot                             | 搜索引擎                  |
| Linked    | linkedinbot                         | 应用内搜索                 |
| 百度        | baiduspider                         | 搜索引擎                  |
| 奇虎 360    | 360Spider                           | 搜索引擎                  |
| 搜狗        | Sogou Spider                        | 搜索引擎                  |
| Yahoo     | Yahoo! Slurp China                  | 搜索引擎                  |
| Yahoo     | Yahoo! Slurp                        | 搜索引擎                  |
| 头条        | Bytespider                          | 搜索引擎                  |
| Twitter   | twitterbot                          | 应用内搜索                 |
| Facebook  | facebookexternalhit                 | 应用内搜索                 |
| -         | rogerbot                            | -                     |
| -         | embedly                             | -                     |
| Quora     | quora link preview                  | -                     |
| -         | showyoubot                          | -                     |
| -         | outbrain                            | -                     |
| -         | pinterest                           | -                     |
| -         | slackbot                            | -                     |
| -         | vkShare                             | -                     |
| -         | W3C_Validator                       | -                     |

nginx 判断：

```conf
if ($http_user_agent ~* "googlebot|bingbot|yandex|baiduspider|360Spider|Sogou Spider|Bytespider|twitterbot|facebookexternalhit|rogerbot|linkedinbot|embedly|quora link preview|showyoubot|outbrain|pinterest\/0\.|pinterestbot|slackbot|vkShare|W3C_Validator|whatsapp") {
## do something
}
```

常见爬虫 UserAgent 汇总

![](https://content.hackertalk.net/pub/WajE7j-B3373R1)

原文：[Every damn time](https://www.reddit.com/r/ProgrammerHumor/comments/og0qk1/every_damn_time/)

Reddit 笑话一则：前端和后端的区别

jwt 真的安全吗？

看到 redis 博客上的一篇文章：[JSON Web Tokens (JWT) are Dangerous for User Sessions—Here’s a Solution](https://redislabs.com/blog/json-web-tokens-jwt-are-dangerous-for-user-sessions/)，有点质疑，大家怎么看呢？

![](https://content.hackertalk.net/pub/Wa083j-BzPL02D)

jwt 真的安全吗？



![](https://image.hackertalk.net/images/cb9632d9-592a-458d-b2d8-ad1fff08d73c)



```tsx
import qs from 'node:querystring';


/**
 npm 的接口 
 https://api.npmjs.org/downloads/point/2023-01-01:2023-12-31/react
 https://github.com/npm/registry/blob/master/docs/download-counts.md

 npmmirror 的接口
 https://github.com/cnpm/cnpmcore/blob/67f1a2476dd1330f518673a1f1f476252a386da1/app/port/controller/DownloadController.ts#L22
*/

// 自定义时间
// let _range = "2023-01-01:2023-12-31";

// 一年
let _range = getFullYearString("2023")

// 切换
// 0:npm
// 1:npmmirror
let getData = 1 ? getNpmmirrorData : getNpmData;

Promise.all([
    // 热门框架
    getData("react"),
    getData("vue"),
    getData("@angular/core"),
    getData("solid-js"),
    getData("svelte"),

    // SSR 框架
    getData("next"),
    getData("nuxt"),

    // react 组件库
    getData("@mui/material"),
    getData("antd"),
    getData("@chakra-ui/react"),
    getData("react-bootstrap"),
    getData("reactstrap"),
    getData("primereact"),
    getData("@mantine/core"),
    getData("@nextui-org/react"),
    getData("shadcn-ui"),
    getData("@radix-ui/react-popover"), // 包太多,随便选一个
    getData("react-vant"),
    getData("@alifd/next"),
    getData("@arco-design/web-react"),
    getData("@arco-design/mobile-react"),
    getData("@douyinfe/semi-ui"),
    getData("tdesign-react"),

    // vue 组件库
    getData("ant-design-vue"),
    getData("element-ui"),
    getData("element-plus"),
    getData("vuetify"),
    getData("iview"),
    getData("view-ui-plus"),
    getData("quasar"),
    getData("naive-ui"),
    getData("@arco-design/web-vue"),
    getData("@opentiny/vue"),
    getData("bootstrap-vue"),

]).then(r => {
    // 从高到低 排序
    r.sort((a, b) => b.downloads - a.downloads);

    let ret = [];
    for (const item of r) {
        ret.push({
            name: item.name,
            downloads: formatNumberToChinese(item.downloads)
        })
    }

    console.table(ret)
})


async function getNpmmirrorData(packageName: string) {
    let host = "https://registry.npmmirror.com/downloads/range";
    let url = `${host}/${_range}/${qs.escape(packageName)}`;
    // console.log(url);
    let downloads = 0;
    let r = await fetch(url).then(r => r.json())
    delete r.versions
    let dataList = r.downloads;
    for (const item of dataList) {
        downloads += item.downloads
    }

    return { name: packageName, downloads: downloads };
}

async function getNpmData(packageName: string) {
    let host = "https://api.npmjs.org/downloads/point";
    let url = `${host}/${_range}/${qs.escape(packageName)}`;
    // console.log(url);
    let r = await fetch(url, { headers: getNpmHeader() })
        .then(r => r.json())
    let downloads = r.downloads;
    return { name: packageName, downloads: downloads };
}

function getFullYearString(year: string) {
    return `${year}-01-01:${year}-12-31`;
}


function getNpmHeader() {
    return new Headers({
        "Accept": 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',
        "Accept-Encoding": "gzip, deflate, br",
        "Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6",
        "Cache-Control": "no-cache",
        "Pragma": "no-cache",
        "Sec-Ch-Ua": 'Chromium";v="122", "Not(A:Brand";v="24", "Microsoft Edge";v="122',
        "Sec-Ch-Ua-Mobile": "?0",
        "Sec-Ch-Ua-Platform": "Windows",
        "Sec-Fetch-Dest": "document",
        "Sec-Fetch-Mode": "navigate",
        "Sec-Fetch-Site": "none",
        "Sec-Fetch-User": "?1",
        "Upgrade-Insecure-Requests": "1",
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0",
    })
}

function formatNumberToChinese(num: number) {
    if (typeof num !== 'number') {
        return '请输入有效的数字';
    }

    if (num >= 10000 && num < 100000000) {
        return (num / 10000).toFixed(0) + '万';
    } else if (num >= 100000000) {
        return (num / 100000000).toFixed(0) + '亿';
    } else {
        return num.toString();
    }
}
```






前端框架与组件库下载量排名

标识符

483DAB9E-B587-4DAE-936C-201F5BC4BCBA

备注

Error Domain=kCFStreamErrorDomainSSL Code=-9806"(null)" UserInfo=/NSLocalizedRecoverySuggestion=Errorcode definition can be found in Apple's SecureTransport.h}

请求信息

CONNECT m.m******(手动打码).pt:443 HTTP/1.1

请求头部

Host: m.m*******.ptConnection: keep-alive

发送请求头部

未发送

服务端开始响应

未响应

请求完成2024/04/26 10:11:40.529

 app无法登录，抓包数据。网页版可以登录，求分析求解

前些天一直在找便宜的云服务器，偶然间发现一免费的云平台，这里推荐给大家。

就是阿贝云，网站是https://www.abeiyun.com

说一下感受

1、阿贝云虽然是一小众的云平台，重在免费，该有的东西都有，还能免备案这是最舒服的。

2、使用了一天，上了项目，网速不慢。

3、唯一要指出来就是磁盘太小。

分享免费云服务器

拉取了rustbuster的v3.0.3版本文件, 二进制文件无法使用, 执行结果如下:

```bash
[root_cn@archlinux target]$ ./rustbuster-v3.0.3-x86_64-unknown-linux-gnu
./rustbuster-v3.0.3-x86_64-unknown-linux-gnu: error while loading shared libraries: libssl.so.1.1: cannot open shared object file: No such file or directory
```

拉取了main源码, 想自行尝试编译, 然后卡在了openssl, 以下为报错信息:

```rust
The following warnings were emitted during compilation:

warning: openssl-sys@0.9.47: build/expando.c:4:24: error: pasting "RUST_VERSION_OPENSSL_" and "(" does not give a valid preprocessing token
warning: openssl-sys@0.9.47:     4 | #define VERSION2(n, v) RUST_VERSION_##n##_##v
warning: openssl-sys@0.9.47:       |                        ^~~~~~~~~~~~~
warning: openssl-sys@0.9.47: build/expando.c:5:23: note: in expansion of macro ‘VERSION2’
warning: openssl-sys@0.9.47:     5 | #define VERSION(n, v) VERSION2(n, v)
warning: openssl-sys@0.9.47:       |                       ^~~~~~~~
warning: openssl-sys@0.9.47: build/expando.c:10:1: note: in expansion of macro ‘VERSION’
warning: openssl-sys@0.9.47:    10 | VERSION(OPENSSL, OPENSSL_VERSION_NUMBER)
warning: openssl-sys@0.9.47:       | ^~~~~~~

error: failed to run custom build command for `openssl-sys v0.9.47`

Caused by:
  process didn't exit successfully: `/mnt/d/work/file/rust/rustbuster/target/debug/build/openssl-sys-c5a482a9e0e49574/build-script-main` (exit status: 101)
  --- stdout
  cargo:rustc-cfg=const_fn
  cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_GNU_OPENSSL_LIB_DIR
  X86_64_UNKNOWN_LINUX_GNU_OPENSSL_LIB_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_LIB_DIR
  OPENSSL_LIB_DIR unset
  cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_GNU_OPENSSL_INCLUDE_DIR
  X86_64_UNKNOWN_LINUX_GNU_OPENSSL_INCLUDE_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_INCLUDE_DIR
  OPENSSL_INCLUDE_DIR unset
  cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_GNU_OPENSSL_DIR
  X86_64_UNKNOWN_LINUX_GNU_OPENSSL_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_DIR
  OPENSSL_DIR unset
  cargo:rustc-link-lib=ssl
  cargo:rustc-link-lib=crypto
  OPT_LEVEL = Some("0")
  TARGET = Some("x86_64-unknown-linux-gnu")
  HOST = Some("x86_64-unknown-linux-gnu")
  CC_x86_64-unknown-linux-gnu = None
  CC_x86_64_unknown_linux_gnu = None
  HOST_CC = None
  CC = None
  CFLAGS_x86_64-unknown-linux-gnu = None
  CFLAGS_x86_64_unknown_linux_gnu = None
  HOST_CFLAGS = None
  CFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("true")
  CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
  running: "cc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-E" "build/expando.c"
  cargo:warning=build/expando.c:4:24: error: pasting "RUST_VERSION_OPENSSL_" and "(" does not give a valid preprocessing token
  cargo:warning=    4 | #define VERSION2(n, v) RUST_VERSION_##n##_##v
  cargo:warning=      |                        ^~~~~~~~~~~~~
  cargo:warning=build/expando.c:5:23: note: in expansion of macro ‘VERSION2’
  cargo:warning=    5 | #define VERSION(n, v) VERSION2(n, v)
  cargo:warning=      |                       ^~~~~~~~
  cargo:warning=build/expando.c:10:1: note: in expansion of macro ‘VERSION’
  cargo:warning=   10 | VERSION(OPENSSL, OPENSSL_VERSION_NUMBER)
  cargo:warning=      | ^~~~~~~
  exit status: 1

  --- stderr
  thread 'main' panicked at /home/root_cn/.cargo/registry/src/mirrors.sjtug.sjtu.edu.cn-4f7dbcce21e258a2/openssl-sys-0.9.47/build/main.rs:126:13:

  Header expansion error:
  Error { kind: ToolExecError, message: "Command \"cc\" \"-O0\" \"-ffunction-sections\" \"-fdata-sections\" \"-fPIC\" \"-g\" \"-fno-omit-frame-pointer\" \"-m64\" \"-Wall\" \"-Wextra\" \"-E\" \"build/expando.c\" with args \"cc\" did not execute successfully (status code exit status: 1)." }

  Failed to find OpenSSL development headers.

  You can try fixing this setting the `OPENSSL_DIR` environment variable
  pointing to your OpenSSL installation or installing OpenSSL headers package
  specific to your distribution:

      # On Ubuntu
      sudo apt-get install libssl-dev
      # On Arch Linux
      sudo pacman -S openssl
      # On Fedora
      sudo dnf install openssl-devel

  See rust-openssl README for more information:

      https://github.com/sfackler/rust-openssl#linux

  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
```

经过检查openssl与rust均为最新版本, 

```bash
[root_cn@archlinux target]$ sudo pacman -S rust
[sudo] password for root_cn:
warning: rust-1:1.77.2-1 is up to date -- reinstalling
resolving dependencies...
looking for conflicting packages...

Package (1)  Old Version  New Version  Net Change

extra/rust   1:1.77.2-1   1:1.77.2-1     0.00 MiB

Total Installed Size:  249.41 MiB
Net Upgrade Size:        0.00 MiB

:: Proceed with installation? [Y/n] n
[root_cn@archlinux target]$ sudo pacman -S openssl
warning: openssl-3.2.1-1 is up to date -- reinstalling
resolving dependencies...
looking for conflicting packages...

Package (1)   Old Version  New Version  Net Change

core/openssl  3.2.1-1      3.2.1-1        0.00 MiB

Total Installed Size:  11.45 MiB
Net Upgrade Size:       0.00 MiB

:: Proceed with installation? [Y/n] n
```

在网上找了一下, 查到资料, 在[rust-openssl的issues](https://github.com/sfackler/rust-openssl/issues/1227)中提到类似报错, 得到解释如下:

> If you are cross compiling to MUSL, you either need to point openssl-sys to the location that you have a copy of OpenSSL cross compiled to MUSL, or tell it to build a copy of OpenSSL itself with the vendored feature.
>
>	如果你是交叉编译到MUSL，你需要将openssl-sys指向你有一个交叉编译到MUSL的OpenSSL副本的位置，或者告诉它用vendored特性构建一个OpenSSL本身的副本。

由于不会rust, 依靠关键词"交叉编译"+"cargo"搜索到了[cargo交叉编译rust程序](https://blog.csdn.net/tyc880616/article/details/104371502), 顺着找到文件:

```
[root_cn@archlinux .cargo]$ cat ~/.cargo/config
[source.crates-io]
replace-with = 'sjtu' # 指定使用下面哪个源，修改为source.后面的内容即可

# 中国科学技术大学
[source.ustc]
registry = "https://mirrors.ustc.edu.cn/crates.io-index"

# 上海交通大学
[source.sjtu]
registry = "https://mirrors.sjtug.sjtu.edu.cn/git/crates.io-index/"

# 清华大学
[source.tuna]
registry = "https://mirrors.tuna.tsinghua.edu.cn/git/crates.io-index.git"

# rustcc社区
[source.rustcc]
registry = "https://code.aliyun.com/rustcc/crates.io-index.git"
```

没有找到交叉编译的设置, 由于本人不会rust, 线索断了, 尝试向大家寻求一波帮助, 非常感谢阅读.



急切寻求rust编译问题帮助(archlinux, openssl)

永久免费，无需注册

[https://good.xjai.top](https://good.xjai.top)

分享免费GPT4站点

[MoienTajik/AspNetCore-Developer-Roadmap (github.com)](https://github.com/MoienTajik/AspNetCore-Developer-Roadmap/blob/master/ReadMe.zh-Hant.md)

![](https://github.com/MoienTajik/AspNetCore-Developer-Roadmap/raw/master/aspnetcore-developer-roadmap.zh-Hant.png)



【轉貼】ASP.NET Core Developer Roadmap in 2024

南京大学数字金融 - 新华报业集团联合诚邀全职博士后！

【研究方向涵盖】

数智传播、数智金融经济、数据金融经济、数智政务、数智监管治理，以及“大数据、大模型、大计算”全新范式等方向理论与实践

 【具体研究领域】

包括 Bigdata4SocialScience、AI4SocialScience、LLM4SocialScience、Blockchain4SocialScience，以及先进信息技术融合创新在数智金融经济、数智传播社会、数智健康医疗、数智政务治理中的应用。

【薪资待遇优渥】

年薪可达 30 万人民币左右，另有绩效奖金及江苏省和南京市人才补贴。具体薪资需与HR面议，总收入依据专业背景和技能经验而定（计算机 AI 背景者薪资更优）。

【欢迎学科交叉、专业背景不限】

不限制专业背景，致力于构建一个多元、开放、包容的科研环境，因此，欢迎计算机、软件、大数据、人工智能、区块链、隐私计算、数据安全、数据科学等背景的博士人才，也热切邀请具有人文社科背景的博士加入我们的团队，共同探索信息技术与社会科学的交叉前沿。也欢迎其他编程能力强的理工科博士毕业生（即使没有AI等背景也欢迎）。

【南京大学研究中心】

南京大学研究中心为双院士领军的校级科研和创新平台，详情请见：fintech.nju.edu.cn。

【团队氛围】

学科交叉、理论与实践相结合、国际化视野、行业前沿视角和项目。

【联系我们】

请感兴趣的朋友将个人简历发送至邮箱:blockchainfintech@126.com。

【更多招募】 南京大学研究中心招收硕士生、博士生、博士后、青年教师、研究员，也欢迎访问、客座、联培以及科研助理（有些岗位可兼职或远程合作）等

欢迎加入我们，您将在这个富有挑战性和机遇的环境中，开启广阔的职业发展道路，实现个人价值与学术梦想的双赢！

期待您的加入，携手共创数智未来！