一、Pulsar安装部署一、下载安装包Pulsar安装包包括三个组件：zookeeper、broker、bookie。下载Pulsar二进制包https://pulsar.apache.org/download/解压压缩包tar-zvxfapache-pulsar-2.9.1-bin.tar.gz2.部署zookeeper2.1修改zookeeper配置创建目录mkdir-pdata/zookeeperecho1>data/zookeeper/myid修改zk配置，文件位于conf/zookeeper.conf#数据目录dataDir=data/zookeeper#日志目录dataLogDir=data/zookeeper/logs#zk集群配置，server.1~nserver.1=127.0.0.1:2888:38882.2backgroundstartzookeeperbin/pulsar-daemonstartzookeeperdoingstartzookeeper...启动zookeeper，登录/Users/admin/work/software_install/apache-pulsar-2.9.1/logs/pulsar-zookeeper-M-C02GL1NTQ05P.log注意：在conf/log4j2.yaml中将immediateFlush设置为true将保证日志事件立即刷新到磁盘。出于性能方面的考虑，默认行为已关闭。通过pulsar-daemonbin/pulsar-daemon管理pulsar组件helpError:noenoughargumentsprovided.Usage:pulsar-daemon(start|stop|restart)wherecommandisoneof:brokerRunabroker服务器bookieRunabookieserverzookeeper运行zookeeper服务器configuration-store运行配置存储服务器websocket运行websocket代理服务器functions-worker运行functionsworkerserverstandalone运行独立Pulsarserviceproxy运行ProxyPulsarservice关闭或重启组件，例如broker、bookie和zookeeper2.3查看zookeeper是否启动成功zookeeper启动日志查看zookeeper进程psaxu|grepzookeeper3.元数据初始化3.1初始化命令说明bin/pulsarinitialize-cluster-metadata\--clusterpulsar-cluster-1\--zookeeper127.0.0.1:2181\--configuration-store127.0.0.1:2181\--web-service-urlhttp://127.0.0.1:8080\--web-service-url-tlshttps://127.0.0.1:8443\--broker-service-urlpulsar://127.0.0.1:6650\--broker-service-url-tlspulsar+ssl://127.0.0.1:6651参数说明参数说明参数说明clusterdefaultclusternamezookeeperlocalcluster使用的zkaddressconfiguration-store是多个集群的全局zk集群地址，并且数据在各个集群之间同步。单机集群的地址和上面的参数zookeeper一样，是web-service-urlBroker的管理流地址，比如创建和删除topic等web-services-url-tlsBroker启用TLS，管理flow使用这个地址broker-service-urlBroker数据流地址，收发消息等broker-service-url-tlsBroker启用TLS，数据流使用这个地址备注：生产环境可以使用域名。3.2查看初始化结果bin/pulsarzookeeper-shell[zk:localhost:2181(CONNECTED)1]ls/[admin,bookies,ledgers,pulsar,stream,zookeeper]4.部署BookKeeper集群4.1配置修改bindAddress=127.0.0.1advertisedAddress=127.0.0.1zkServers=127.0.0.1:2181参数说明参数说明bindAddress服务监听的地址，默认为0.0.0.0advertisedAddress服务发布的主机名或IP，默认为InteAddress.getLocalHost().getHostNamezkServerszk集群地址，可以对照Brokershare4.2命令启动bin/pulsar-daemonstartbookiedoingstartbookie...启动bookie，登录/Users/admin/work/software_install/apache-pulsar-2.9.1/logs/pulsar-bookie-M-C02GL1NTQ05P.logNote：在conf/log4j2.yaml中将immediateFlush设置为true将保证日志事件立即刷新到磁盘。出于性能方面的考虑，默认行为已关闭。4.3测试bookie集群bin/bookkeepershellsimpletest--ensemble1--writeQuorum1--ackQuorum1--numEntries1000...2022-02-19T23:43:03,391+0800[main]INFOorg.apache.bookkeeper.tools.cli.commands.client.SimpleTestCommand-722个条目swritten2022-02-19T23:43:03,983+0800[main]INFOorg.apache.bookkeeper.tools.cli.commands.client.SimpleTestCommand-1000个条目写入分类账02022-02-19T23:43:04,041+0800[main]]]INFOorg.apache.bookkeeper.proto.PerChannelBookieClient-Closingtheperchannelbookieclientfor127.0.0.1:3181...备注：通过simpletest命令向bookie集群写入测试数据，完成后自动删除测试完成5.部署Broker集群5.1修改配置zookeeperServers=127.0.0.1:2181configurationStoreServers=127.0.0.1:2181bindAddress=127.0.0.1#默认InetAddress.getLocalHost().getHostName()advertisedAddress=127.0.0.1clusterNameluster2=pulsoker.binstart-pulsoker-cbin/pulsar-daemonstartbrokerdoingstartbroker...startingbroker,loggingto/Users/admin/work/software_install/apache-pulsar-2.9.1/logs/pulsar-broker-M-C02GL1NTQ05P.log注意：设置immediateFlush为conf/log4j2.yaml中的true将保证日志记录事件立即刷新到磁盘。出于性能方面的考虑，默认行为已关闭。5.3验证集群，查看集群节点bin/pulsar-adminbrokerslistcluster-1"172.17.13.184:8080"发送测试消息bin/pulsar-clientproducepersistent://public/default/test-n1-m"你好Pulsar"...2022-02-20T13:31:18,469+0800[main]INFOorg.apache.pulsar.client.cli.PulsarClientTool-成功生成1条消息...使用测试消息bin/pulsar-client使用持久://public/default/test-n100-s"consumer-test"-t"Exclusive"...-----gotmessage-----key:[null],properties:[],content:HelloPulsar...Summary:至此测试集群搭建完成，下面将介绍生产环境配置的调整项2.生产环境配置1.机器配置下面是生产环境的Pulsar集群，由3个zookeeper节点，3个broker节点，5个bookie节点组成组件配置组件配置zookeeper4C8G100G*3broker16C64G500G*3bookie16C64G500G*5备注：各组件集群部署时，可以跨同城可用区部署，提高高可用，broker不需要存储100G的消息，bookie通常需要更大的磁盘，比如3T来存储消息，根据消息的多少来计算。2.内存优化配置项的内存大小或比例，系统OS缓存总大小为1~2G，Jvm内存和堆外内存的1/2（去除系统缓存后剩余缓存的一半）)，其中Jvm堆内存占1/3，堆外内存DirectMemory占2/3PageCache内存大小1/2（去除系统缓存后剩余缓存的一半）2.1Broker内存调整为64G内存大小，修改conf/pulsar_env.sh文件中如下内容：PULSAR_MEM=${PULSAR_MEM:-"-Xms10g-Xmx10g-XX:MaxDirectMemorySize=20g"}2.2Bookie内存调整为64G内存大小，修改文件中如下内容conf/bkenv.sh:BOOKIE_MEM=${BOOKIE_MEM:-${PULSAR_MEM:-"-Xms10g-Xmx10g-XX:MaxDirectMemorySize=20g"}}3.Broker调整项配置项说明zookeeperServers=x.x.x.x:2181,x.x.x.x:2181,x.x.x.x:2181本地zookeeper集群地址配置StoreServers=x.x.x.x:2181,x.x.x.x:2181,x.x.x.xindsconfigurationstorageaddress:Z.x.x.xinds1clusteraddress=x.x.x.x服务监听的地址可以是本地IP，默认为0.0.0.0advertisedAddress=x.x.x.x服务发布的主机名或IP，默认为IntetAddress.getLocalHost().getHostNameclusterName=cluster-xxxclusternamebrokerDeleteInactiveTopicsEnabled=falseoff自动删除不活跃的topicdefaultNumberOfNamespaceBundles=12Bundles的数量应该是broker数量的整数倍，默认是4触发删除策略backlogQuotaDefaultLimitGB=-1保持默认，未被消费确认的消息消息存储大小默认为-1，即没有限制。可以通过set-message-ttl设置过期时间，防止磁盘被占满指定Ledgermanaged时Ensemble的大小LedgerDefaultWriteQuorum=3指定创建Ledger时Quorum的大小managedLedgerDefaultAckQuorum=2指定创建Ledger时ackQuorum的大小dispatcherMaxReadBatchSize=500一次读取bookkeeper的数量，默认100loadBalancerAutoBundleSplitEnabled=false关闭自动bundle拆分功能，提高客户端稳定性loadBalancerAutoUnloadSplitBundlesEnabled=false关闭自动bundle拆分功能，提高客户端稳定性loadBalancerSheddingEnabled=false禁止Pulsar自动平衡loadBalancerEnabled=false禁止Pulsar自动平衡调整。在线启用负载均衡时，发现有重复的消息。先在这里关闭。4.Bookie调整项配置项说明配置项说明zkServers=x.x.x.x:2181,x.x.x.x:2181,x.x.x.x:2181本地zookeeper集群地址journalDirectory=/data/bookkeeper/journal01,/data/bookkeeper/journal02BookKeeper存放其pre-的目录写入的日志，多个目录用逗号隔开，防止线程阻塞多个ledger目录需要用逗号隔开注意：如果条件允许，journalDirectory和ledgerDirectories可以配置在不同的磁盘上。本文转载自微信公众号“瓜农老粮”，可通过以下二维码关注。转载本文请联系瓜农老梁公众号。
