Nest9.0系列-09 - pulsar-client安装与Serverless局限性探讨 - 玩转Nestjs

接上篇博文，如何向 TDMQ 的 Topic 发送信息

我们上次采用了 Pulsar 作为 TDMQ 的消息列队服务，相应地，操作它的就是 pulsar-client 这个库了

由于请求只能在内网中，所以我们只能在云服务器中安装，云服务器是 Centos7.6 的系统

以下是安装记录

# 安装 `pulsar-client`

官方文档上介绍安装 nodejs 版的 pulsar-client 只需要如下语句

$ npm install pulsar-client -S

执行之后毫无意外的报错了，遂查阅了 N 多资料，最后在 pulsar-client-node github 项目里才知道原来安装它之前需要安装一系列软件与 C++ 库

# Centos 安装 `Pulsar` C++ 库

https://github.com/apache/pulsar-client-node

yum 安装 gcc-c++,make

$ yum install gcc-c++ make

下载 rpm 包

wget https://archive.apache.org/dist/pulsar/pulsar-2.9.1/RPMS/apache-pulsar-client-2.9.1-1.x86_64.rpm

wget https://archive.apache.org/dist/pulsar/pulsar-2.9.1/RPMS/apache-pulsar-client-devel-2.9.1-1.x86_64.rpm

安装 rpm 包

$ rpm -ivh apache-pulsar-client*.rpm

最后再执行 npm install

$ npm install pulsar-client -S

# 使用 `Pulsar`

编写 index.js 测试

	const Pulsar = require("pulsar-client");

	(async () => {
	const client = new Pulsar.Client({
	serviceUrl: "http://pulsar-jm4bd43m2jab.tdmq-pulsar.ap-sh.qcloud.tencenttdmq.com:5037",
	authentication: new Pulsar.AuthenticationToken({
	token: "eyJrZXlJZCI6Ixxxxxx", // 更换为密钥
	}),
	});

	const producer = await client.createProducer({
	topic: 'pulsar-jm4bdxxxxx/sendmail/sendmail', // or 'my-tenant/my-namespace/my-topic' to specify topic's tenant and namespace
	});
	let data = {
	user:"测试",
	email:"1015517471@qq.com"
	};
	await producer.send({
	data: Buffer.from(JSON.stringify(data)),
	});

	await client.close();
	})();

根据官网文档及腾讯云关于 Pulsar 文档介绍

秘钥可以从上篇文章中角色管理中获取

接下来使用 node 执行 index.js

	➜ /home/t node index.js
	/home/t/node_modules/bindings/bindings.js:121
	throw e;
	^

	Error: libpulsar.so.2.9.1: cannot open shared object file: No such file or directory
	at Object.Module._extensions..node (node:internal/modules/cjs/loader:1183:18)
	at Module.load (node:internal/modules/cjs/loader:981:32)
	at Function.Module._load (node:internal/modules/cjs/loader:822:12)
	at Module.require (node:internal/modules/cjs/loader:1005:19)
	at require (node:internal/modules/cjs/helpers:102:18)
	at bindings (/home/t/node_modules/bindings/bindings.js:112:48)
	at Object.<anonymous> (/home/t/node_modules/pulsar-client/index.js:20:42)
	at Module._compile (node:internal/modules/cjs/loader:1101:14)
	at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
	at Module.load (node:internal/modules/cjs/loader:981:32) {
	code: 'ERR_DLOPEN_FAILED'
	}

又报错了... 错误信息说明 libpulsar.so.2.9.1 找不到

经过查找本服务器是可以找到 libpulsar.so.2.9.1 的，在 /usr/lib/ 下，猜测是不是默认路径不对？也许在 /usr/lib64/ 下才是正确的？

建立一条软连接

$ ln -s /usr/lib/libpulsar.so.2.9.1 /usr/lib64/libpulsar.so.2.9.1

再次执行 index.js

	$ node index.js
	2022-08-31 11:40:39.043 INFO [140338895238912] HTTPLookupService:235 \| Curl Lookup Request sent for http://pulsar-jm4bd43m2jab.tdmq-pulsar.ap-sh.qcloud.tencenttdmq.com:5039/admin/v2/persistent/pulsar-jm4bd43m2jab/sendmail/sendmail/partitions?checkAllowAutoCreation=true
	2022-08-31 11:40:39.155 INFO [140338895238912] HTTPLookupService:249 \| Response received for url http://pulsar-jm4bd43m2jab.tdmq-pulsar.ap-sh.qcloud.tencenttdmq.com:5039/admin/v2/persistent/pulsar-jm4bd43m2jab/sendmail/sendmail/partitions?checkAllowAutoCreation=true code 200
	2022-08-31 11:40:39.155 INFO [140338895238912] HTTPLookupService:293 \| parsePartitionData = { LookupDataResult [brokerUrl_ = ] [brokerUrlTls_ = ] [partitions = 2] [authoritative = 0] [redirect = 0] proxyThroughServiceUrl = 0] }
	2022-08-31 11:40:39.160 INFO [140338895238912] HandlerBase:64 \| [persistent://pulsar-jm4bd43m2jab/sendmail/sendmail-partition-0, ] Getting connection from pool
	2022-08-31 11:40:39.160 INFO [140338895238912] HandlerBase:64 \| [persistent://pulsar-jm4bd43m2jab/sendmail/sendmail-partition-1, ] Getting connection from pool
	2022-08-31 11:40:39.160 INFO [140338895238912] HTTPLookupService:235 \| Curl Lookup Request sent for http://pulsar-jm4bd43m2jab.tdmq-pulsar.ap-sh.qcloud.tencenttdmq.com:5039/lookup/v2/topic/persistent/pulsar-jm4bd43m2jab/sendmail/sendmail-partition-0
	2022-08-31 11:40:39.187 INFO [140338895238912] HTTPLookupService:249 \| Response received for url http://pulsar-jm4bd43m2jab.tdmq-pulsar.ap-sh.qcloud.tencenttdmq.com:5039/lookup/v2/topic/persistent/pulsar-jm4bd43m2jab/sendmail/sendmail-partition-0 code 200
	2022-08-31 11:40:39.187 INFO [140338895238912] HTTPLookupService:328 \| parseLookupData = { LookupDataResult [brokerUrl_ = pulsar://169.254.0.168:5035] [brokerUrlTls_ = ] [partitions = 0] [authoritative = 0] [redirect = 0] proxyThroughServiceUrl = 0] }
	2022-08-31 11:40:39.187 INFO [140338895238912] ClientConnection:181 \| [<none> -> pulsar://169.254.0.168:5035] Create ClientConnection, timeout=10000
	2022-08-31 11:40:39.187 INFO [140338895238912] ConnectionPool:96 \| Created connection for pulsar://169.254.0.168:5035
	2022-08-31 11:40:39.188 INFO [140338895238912] HTTPLookupService:235 \| Curl Lookup Request sent for http://pulsar-jm4bd43m2jab.tdmq-pulsar.ap-sh.qcloud.tencenttdmq.com:5039/lookup/v2/topic/persistent/pulsar-jm4bd43m2jab/sendmail/sendmail-partition-1
	2022-08-31 11:40:39.193 INFO [140338876249856] ClientConnection:367 \| [172.17.16.13:59602 -> 169.254.0.168:5035] Connected to broker
	2022-08-31 11:40:39.201 INFO [140338895238912] HTTPLookupService:249 \| Response received for url http://pulsar-jm4bd43m2jab.tdmq-pulsar.ap-sh.qcloud.tencenttdmq.com:5039/lookup/v2/topic/persistent/pulsar-jm4bd43m2jab/sendmail/sendmail-partition-1 code 200
	2022-08-31 11:40:39.201 INFO [140338895238912] HTTPLookupService:328 \| parseLookupData = { LookupDataResult [brokerUrl_ = pulsar://169.254.0.168:10026] [brokerUrlTls_ = ] [partitions = 0] [authoritative = 0] [redirect = 0] proxyThroughServiceUrl = 0] }
	2022-08-31 11:40:39.202 INFO [140338895238912] ClientConnection:181 \| [<none> -> pulsar://169.254.0.168:10026] Create ClientConnection, timeout=10000
	2022-08-31 11:40:39.202 INFO [140338895238912] ConnectionPool:96 \| Created connection for pulsar://169.254.0.168:10026
	2022-08-31 11:40:39.202 INFO [140338876249856] ProducerImpl:188 \| [persistent://pulsar-jm4bd43m2jab/sendmail/sendmail-partition-0, ] Created producer on broker [172.17.16.13:59602 -> 169.254.0.168:5035]
	2022-08-31 11:40:39.204 INFO [140338876249856] ClientConnection:367 \| [172.17.16.13:46318 -> 169.254.0.168:10026] Connected to broker
	2022-08-31 11:40:39.216 INFO [140338876249856] ProducerImpl:188 \| [persistent://pulsar-jm4bd43m2jab/sendmail/sendmail-partition-1, ] Created producer on broker [172.17.16.13:46318 -> 169.254.0.168:10026]
	2022-08-31 11:40:39.233 INFO [140338903631616] ClientImpl:492 \| Closing Pulsar client with 1 producers and 0 consumers
	2022-08-31 11:40:39.233 INFO [140338903631616] ProducerImpl:621 \| [persistent://pulsar-jm4bd43m2jab/sendmail/sendmail-partition-0, tdmq_sh_pulsar_release-269-8963064] Closing producer for topic persistent://pulsar-jm4bd43m2jab/sendmail/sendmail-partition-0
	2022-08-31 11:40:39.233 INFO [140338903631616] ProducerImpl:621 \| [persistent://pulsar-jm4bd43m2jab/sendmail/sendmail-partition-1, tdmq_sh_pulsar_release-258-12777610] Closing producer for topic persistent://pulsar-jm4bd43m2jab/sendmail/sendmail-partition-1
	2022-08-31 11:40:39.235 INFO [140338876249856] ProducerImpl:664 \| [persistent://pulsar-jm4bd43m2jab/sendmail/sendmail-partition-0, tdmq_sh_pulsar_release-269-8963064] Closed producer
	2022-08-31 11:40:39.236 INFO [140338876249856] ProducerImpl:664 \| [persistent://pulsar-jm4bd43m2jab/sendmail/sendmail-partition-1, tdmq_sh_pulsar_release-258-12777610] Closed producer
	2022-08-31 11:40:39.236 INFO [140338876249856] ClientConnection:1535 \| [172.17.16.13:46318 -> 169.254.0.168:10026] Connection closed
	2022-08-31 11:40:39.236 INFO [140338876249856] ClientConnection:1535 \| [172.17.16.13:59602 -> 169.254.0.168:5035] Connection closed

没有错误，而且打印出了很多 INFO 信息，去云函数日志查看，果然有一条调用信息，接下来我陷入深深的思考...

pulsar-client 在运行时也依赖于当前环境下的 libpulsar.so.2.9.1 , 那么如果换个环境是不是就因为无法找到该文件而报错了

# Serverless 的局限性

虽然在我的云服务器将 pulsar-client 安装成功，也成功运行起来了，但是其过程之曲折，调试之麻烦可见一斑，如果我们要把它部署到 Serverless 上呢？我没有尝试.

各个应用程序有时候会依赖一些环境本身的库，甚至是操作系统底层的一些东西，这方面 PHP 尤其明显，Nodejs 有少部分也有这种情况， pulsar-client 就是一个典型的例子

如果我们的应用程序采用 Serverless 部署，需要参考 Serverless 提供商提供的环境，例如腾讯云 Serverless 的 PHP 环境安装第三方扩展的时候就及其麻烦，如果发生实在安装不了的扩展，但是项目必须要用到的时候，需要切换回 Docker 的部署方式，那就牺牲了 Serverless 的便利性和经济性，毕竟我们选择 Serverless 的初衷是希望轻量且经济，这就是 Serverless 的局限性.

# 解决办法

好在 pulsar-client 不是唯一操作 TDMQ 消息队列的方式，还可以使用腾讯云本身的 SDK

安装 tencentcloud-sdk-nodejs

$ npm install tencentcloud-sdk-nodejs-tdmq --save

	// Depends on tencentcloud-sdk-nodejs version 4.0.3 or higher
	const tencentcloud = require("tencentcloud-sdk-nodejs");

	const TdmqClient = tencentcloud.tdmq.v20200217.Client;

	// 实例化一个认证对象，入参需要传入腾讯云账户 secretId，secretKey, 此处还需注意密钥对的保密
	// 密钥可前往 https://console.cloud.tencent.com/cam/capi 网站进行获取
	const clientConfig = {
	credential: {
	secretId: "SecretId",
	secretKey: "SecretKey",
	},
	region: "ap-shanghai",
	profile: {
	httpProfile: {
	endpoint: "tdmq.tencentcloudapi.com",
	},
	},
	};

	// 实例化要请求产品的 client 对象，clientProfile 是可选的
	const client = new TdmqClient(clientConfig);
	const params = {
	"Topic": "pulsar-jm4bd43m2jab/sendmail/sendmail",
	"Payload": "{\"user\":\"hello\",\"email\":\"1015517471@qq.com\"}"
	};
	client.SendMessages(params).then(
	(data) => {
	console.log(data);
	},
	(err) => {
	console.error("error", err);
	}
	);

	$ node index.js
	2022-08-31 11:40:39.043 INFO [140338895238912] HTTPLookupService:235 \| Curl Lookup Request sent for http://pulsar-jm4bd43m2jab.tdmq-pulsar.ap-sh.qcloud.tencenttdmq.com:5039/admin/v2/persistent/pulsar-jm4bd43m2jab/sendmail/sendmail/partitions?checkAllowAutoCreation=true
	2022-08-31 11:40:39.155 INFO [140338895238912] HTTPLookupService:249 \| Response received for url http://pulsar-jm4bd43m2jab.tdmq-pulsar.ap-sh.qcloud.tencenttdmq.com:5039/admin/v2/persistent/pulsar-jm4bd43m2jab/sendmail/sendmail/partitions?checkAllowAutoCreation=true code 200
	2022-08-31 11:40:39.155 INFO [140338895238912] HTTPLookupService:293 \| parsePartitionData = { LookupDataResult [brokerUrl_ = ] [brokerUrlTls_ = ] [partitions = 2] [authoritative = 0] [redirect = 0] proxyThroughServiceUrl = 0] }
	2022-08-31 11:40:39.160 INFO [140338895238912] HandlerBase:64 \| [persistent://pulsar-jm4bd43m2jab/sendmail/sendmail-partition-0, ] Getting connection from pool
	2022-08-31 11:40:39.160 INFO [140338895238912] HandlerBase:64 \| [persistent://pulsar-jm4bd43m2jab/sendmail/sendmail-partition-1, ] Getting connection from pool
	2022-08-31 11:40:39.160 INFO [140338895238912] HTTPLookupService:235 \| Curl Lookup Request sent for http://pulsar-jm4bd43m2jab.tdmq-pulsar.ap-sh.qcloud.tencenttdmq.com:5039/lookup/v2/topic/persistent/pulsar-jm4bd43m2jab/sendmail/sendmail-partition-0
	2022-08-31 11:40:39.187 INFO [140338895238912] HTTPLookupService:249 \| Response received for url http://pulsar-jm4bd43m2jab.tdmq-pulsar.ap-sh.qcloud.tencenttdmq.com:5039/lookup/v2/topic/persistent/pulsar-jm4bd43m2jab/sendmail/sendmail-partition-0 code 200
	2022-08-31 11:40:39.187 INFO [140338895238912] HTTPLookupService:328 \| parseLookupData = { LookupDataResult [brokerUrl_ = pulsar://169.254.0.168:5035] [brokerUrlTls_ = ] [partitions = 0] [authoritative = 0] [redirect = 0] proxyThroughServiceUrl = 0] }
	2022-08-31 11:40:39.187 INFO [140338895238912] ClientConnection:181 \| [<none> -> pulsar://169.254.0.168:5035] Create ClientConnection, timeout=10000
	2022-08-31 11:40:39.187 INFO [140338895238912] ConnectionPool:96 \| Created connection for pulsar://169.254.0.168:5035
	2022-08-31 11:40:39.188 INFO [140338895238912] HTTPLookupService:235 \| Curl Lookup Request sent for http://pulsar-jm4bd43m2jab.tdmq-pulsar.ap-sh.qcloud.tencenttdmq.com:5039/lookup/v2/topic/persistent/pulsar-jm4bd43m2jab/sendmail/sendmail-partition-1
	2022-08-31 11:40:39.193 INFO [140338876249856] ClientConnection:367 \| [172.17.16.13:59602 -> 169.254.0.168:5035] Connected to broker
	2022-08-31 11:40:39.201 INFO [140338895238912] HTTPLookupService:249 \| Response received for url http://pulsar-jm4bd43m2jab.tdmq-pulsar.ap-sh.qcloud.tencenttdmq.com:5039/lookup/v2/topic/persistent/pulsar-jm4bd43m2jab/sendmail/sendmail-partition-1 code 200
	2022-08-31 11:40:39.201 INFO [140338895238912] HTTPLookupService:328 \| parseLookupData = { LookupDataResult [brokerUrl_ = pulsar://169.254.0.168:10026] [brokerUrlTls_ = ] [partitions = 0] [authoritative = 0] [redirect = 0] proxyThroughServiceUrl = 0] }
	2022-08-31 11:40:39.202 INFO [140338895238912] ClientConnection:181 \| [<none> -> pulsar://169.254.0.168:10026] Create ClientConnection, timeout=10000
	2022-08-31 11:40:39.202 INFO [140338895238912] ConnectionPool:96 \| Created connection for pulsar://169.254.0.168:10026
	2022-08-31 11:40:39.202 INFO [140338876249856] ProducerImpl:188 \| [persistent://pulsar-jm4bd43m2jab/sendmail/sendmail-partition-0, ] Created producer on broker [172.17.16.13:59602 -> 169.254.0.168:5035]
	2022-08-31 11:40:39.204 INFO [140338876249856] ClientConnection:367 \| [172.17.16.13:46318 -> 169.254.0.168:10026] Connected to broker
	2022-08-31 11:40:39.216 INFO [140338876249856] ProducerImpl:188 \| [persistent://pulsar-jm4bd43m2jab/sendmail/sendmail-partition-1, ] Created producer on broker [172.17.16.13:46318 -> 169.254.0.168:10026]
	2022-08-31 11:40:39.233 INFO [140338903631616] ClientImpl:492 \| Closing Pulsar client with 1 producers and 0 consumers
	2022-08-31 11:40:39.233 INFO [140338903631616] ProducerImpl:621 \| [persistent://pulsar-jm4bd43m2jab/sendmail/sendmail-partition-0, tdmq_sh_pulsar_release-269-8963064] Closing producer for topic persistent://pulsar-jm4bd43m2jab/sendmail/sendmail-partition-0
	2022-08-31 11:40:39.233 INFO [140338903631616] ProducerImpl:621 \| [persistent://pulsar-jm4bd43m2jab/sendmail/sendmail-partition-1, tdmq_sh_pulsar_release-258-12777610] Closing producer for topic persistent://pulsar-jm4bd43m2jab/sendmail/sendmail-partition-1
	2022-08-31 11:40:39.235 INFO [140338876249856] ProducerImpl:664 \| [persistent://pulsar-jm4bd43m2jab/sendmail/sendmail-partition-0, tdmq_sh_pulsar_release-269-8963064] Closed producer
	2022-08-31 11:40:39.236 INFO [140338876249856] ProducerImpl:664 \| [persistent://pulsar-jm4bd43m2jab/sendmail/sendmail-partition-1, tdmq_sh_pulsar_release-258-12777610] Closed producer
	2022-08-31 11:40:39.236 INFO [140338876249856] ClientConnection:1535 \| [172.17.16.13:46318 -> 169.254.0.168:10026] Connection closed
	2022-08-31 11:40:39.236 INFO [140338876249856] ClientConnection:1535 \| [172.17.16.13:59602 -> 169.254.0.168:5035] Connection closed

# 安装 pulsar-client

# Centos 安装 Pulsar C++ 库

# 使用 Pulsar

# Serverless 的局限性

# 解决办法

Nest9.0系列-08 - Serverless事件触发器函数与Nest独立应用

Nest9.0系列-10 - Nest框架集成socket.io来接收websocket请求-搭建简单的聊天室应用

# 安装 `pulsar-client`

# Centos 安装 `Pulsar` C++ 库

# 使用 `Pulsar`