背景:
前段时间,发生了某个服务节点发生问题,被阻塞时,影响了整套系统的运行。期望在运行的时候,当某个节点发生问题的时候,上级服务可以调用另一个可以节点,不会影响整套系统。
分析:
主要是不可用节点阻塞达30秒以上,我们使用的是random loadbalance的负载均衡策略。上级应用不断的调用不可用节点,导致被阻塞的调用越来越多,一直到报线程用完错误。
猜想可解决的方案:
1. 对于一些耗时少的服务,上级应用应该设置比较短的timeout时间,然后设置retry为下级节点数,这样就可以无失败次数
2.对于耗时长的服务,上级应用需要设置长的timeout时间,这时可以设置loadbalance为leastactivity
配置示例:
<dubbo:reference id="helloWorldService" interface="IHelloWorldService" group="test" timeout="5000" loadbalance="leastactive" actives="200"/>
在Consumer上需要添加loadbalance和actives, actives表示Activity最大的并发数,如果超过这个并发数,在invoke时将会等待。Activity的代码如下所示:
int max = invoker.getUrl().getMethodParameter(methodName, Constants.ACTIVES_KEY, 0); RpcStatus count = RpcStatus.getStatus(invoker.getUrl(), invocation.getMethodName()); if (max > 0) { long timeout = invoker.getUrl().getMethodParameter(invocation.getMethodName(), Constants.TIMEOUT_KEY, 0); long start = System.currentTimeMillis(); long remain = timeout; int active = count.getActive(); if (active >= max) { synchronized (count) { while ((active = count.getActive()) >= max) { try { count.wait(remain); } catch (InterruptedException e) { } long elapsed = System.currentTimeMillis() - start; remain = timeout - elapsed; if (remain <= 0) { throw new RpcException("Waiting concurrent invoke timeout in client-side for service: " + invoker.getInterface().getName() + ", method: " + invocation.getMethodName() + ", elapsed: " + elapsed + ", timeout: " + timeout + ". concurrent invokes: " + active + ". max concurrent invoke limit: " + max); } } } } }
测试:
案例1: Consumer 设置为5秒超时,每100毫秒,并发发送100个请求,总共执行150次,预计15秒测试完成。loadbalance策略为Random. Retry次数为0.
Provider 1:每个请求sleep 10毫秒
Provider 2:每个请求sleep 10秒
结果:
Consumer: 总调用次数:15000,成功次数:7460,错误次数:7540,耗时:131227毫秒
Provider1: 执行总数为7460
Provider2: 执行总数为2590,其他请求都报“Thread pool is EXHAUSTED”异常
案例2: Consumer 设置为5秒超时,每100毫秒,并发发送100个请求,总共执行150次,loadbalance策略为Random. Retry次数为1.
Provider 1:每个请求sleep 10毫秒
Provider 2:每个请求sleep 10秒
结果:
Consumer: 总调用次数:15000,成功次数:15000,错误次数:0,耗时:129134毫秒
Provider1: 执行总数为15000
Provider2: 执行总数为2600
案例3: Consumer 设置为5秒超时,每100毫秒,并发发送100个请求,总共执行150次,loadbalance策略为LeastActivity. Retry次数为1.
Provider 1:每个请求sleep 10毫秒
Provider 2:每个请求sleep 10秒
结果:
Consumer: 总调用次数:15000,成功次数:15000,错误次数:0,耗时:17975毫秒
Provider1: 执行总数为15000
Provider2: 执行总数为308
分析:案例1中错误总数和成功总数差不多一致,因为retry为0,50%请求发送到
provider2,发送错误的时候未重试
案例2中错误总数为0,但是耗时明显增长,花费了2分钟时间处理15000次请求。QPS下降。
案例3中错误总数为0,耗时为17秒,和预计的相近。
总结:leastactivity的loadbalance可以在发生故障的时候,保证QPS和可靠性。但是配置了leastactivity之后,权重的配置将会失效。建议配置重试次数,对于时耗短的应用,上级应用的超时时间要配置尽量短。
相关推荐
一款比较好的 asm,对于汇编的学习很有帮助 而且操作简单
solving the least square problems
recursive least square and least mean square
迭代(重)加权最小二乘 论文 Describes a powerful optimization algorithm which iteratively solves a weighted least squares approximation problem in order to solve an L_p approximation problem
Least Squares using c
matlab code for least sqauare
non_linear least square problems 非线性优化讲解的比较好的资料
资源名:libPLS for Partial Least Squares总算法matlab代码_matlab 资源类型:matlab项目全套源码 源码说明: 全部项目源码都是经过测试校正后百分百成功运行的,如果您下载后不能运行可联系我进行指导或者更换。 ...
Solving the global method of Weighted Least Squares (WLS) model in image filtering is both time- and memory- consuming. In this paper, we present an alternative ap- proximation in a time- and memory- ...
2002 Efron Least Angle regression.pdf
Partial Least Squares (PLS) Analysis was first developed in the late 60’s by Herman Wold, and works on the assumption that the focus of analysis is on which aspects of the signal in one matrix are ...
a good material. you can get the best description about least squares
20. Least Angle Regression - The Annals of Statistics
Generalized Least Squares -- good book for least squares
递归最小二乘算法recursive least squareRLS
This book focuses on Least Squares Support Vector Machines (LS-SVMs) which are reformulations to standard SVMs. LS-SVMs are closely related to regularization networks and Gaussian processes but ...
MLS算法论文,We provide an image deformation method based on Moving LeastSquares using various classes of linear functions including affine,similarity and rigid transformations. These deformationsare ...
偏最小二乘partial least squares(PLS)的matlab代码实现,整理为matlab函数,方便实用。
Leastsquares estimation of anisotropic similarity transformations from corresponding 2D point sets
Matlab code for least square estimation