We can refer to the link http://spark.incubator.apache.org/docs/latest/tuning.html for detail tuning document.
After tuning, spark can process 200M logs every minutes in one single work with 1G memory. The time for finish process logs is about 35-40 seconds every duration.
There is some points I do the tuning.
I change the logs, every use conation 20KB logs, so 5000 uses will have 100M logs. When the logs of every use is less, the concurrent process speed will have improvement
JavaDStream<String> stringStream = jsc.socketTextStream("0.0.0.0", ConfigUtil.getInt(ConfigUtil.KEY_SPARK_REMOTE_FLUME_LISTENER_PORT), StorageLevel.MEMORY_AND_DISK())
Add StorageLevel.MEMORY_AND_DISK() when create stream, the default is StorageLevel.MEMORY_AND_DISK2(), will use double memory.
Replace code String user = currentLine.substring(start,end); with String user = new String(currentLine.substring(start,end).toCharArray());
Because substring will create a new String, it will use the same char array with currentLine, thus when some object have refer to user object and currentLine is useless, currentLine also can’t be recycle.
update Spark configure. Below is the latest configuration, when the data is different, sometimes we need update it.
sparkConf.setMaster(ConfigUtil.getString(ConfigUtil.KEY_SPARK_REMOTE_MASTER)).setAppName(appName)
.setJars(new String[]{ConfigUtil.getString(ConfigUtil.KEY_SPARK_REMOTE_JAR_LOCATION)})
.set("spark.executor.memory", "1024m")
.set("spark.streaming.unpersist", "true")
.set("spark.rdd.compress", "true")
.set("spark.default.parallelism", "12")
.set("spark.storage.memoryFraction", "0.3")
.set("spark.cleaner.ttl", "1200")
//.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
// .set("spark.kryo.registrator", "com.seven.oi.spark.KryoRegistratorEx")
//.setExecutorEnv("SPARK_JAVA_OPTS","-XX:NewRatio=1 -XX:+UseCompressedStrings -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps")
//.setExecutorEnv("SPARK_JAVA_OPTS", "-XX:NewRatio=1 -XX:+UseCompressedStrings")
//use fastutil to store map
.set("spark.shuffle.memoryFraction", "0.3");
set spark.rdd.compress = true, thus can use less memory
when out of memory happen, we can also consider increment spark.default.parallelism.
sometimes we can use org.apache.spark.serializer.KryoSerializer, Kryo is significantly faster and more compact than Java serialization. default is Java serialization. (Need more test, current I can’t find any different in my test).
When there is a lot of young GC, we can consider increment the Young generation. Set -XX:NewRatio
When the JDK is 64 bit, we can use -XX:+UseCompressedStrings to compress String. I found JDK with 64 bit will use more memory then JDK with 32 bit.
We also can consider increment spark.shuffle.memoryFraction when have a lot of shuffle operation
If you want spark submit more tasks every time you can increment SPARK_WORKER_CORES. spark will submit tasks less than SPARK_WORKER_CORES every time
相关推荐
apache-spark-best-practices-and-tuning apache-spark-best-practices-and-tuning
VITS-fast-fine-tuning训练准备的样例数据,可以快速体验该模型的语音合成效果。内容包含预训练模型、配置文件、语音素材等。
整理了chatgpt微调所需要的测试资源数据,按照fine-tuning的格式生成的数据集。可以直接 fine_tunes.create -t 使用。 微调完毕后可以 -p 回答结果。 其中包含三四百条数据。 适合测试openai自定义模型的人群使用。
VITS-fast-fine-tuning训练准备的样例数据,可以快速体验该模型的语音合成效果。内容包含预训练模型、配置文件、语音素材等。
Digital Self-tuning Controllers PDF版。需要的可以下载。
The Java Platform, Standard Edition HotSpot Virtual Machine Garbage Collection Tuning Guide describes the garbage collection methods included in the Java HotSpot Virtual Machine (Java HotSpot VM) and ...
微调fine-tuning.pdf微调fine-tuning.pdf微调fine-tuning.pdf微调fine-tuning.pdf
is self-tuning and, therefore, cannot adapt to changing UDF execution patterns. This paper addresses this problem by introducing a self-tuning cost modeling approach based on the quadtree. The ...
藏经阁-Tuning Apache Spark for Large Scale Workloads.pdf
qorvo-antenna-aperture-tuning-eguide_cn.pdf
Speedemy-MySQL-Configuration-Tuning-Handbook.pdf Speedemy-MySQL-Configuration-Tuning-Handbook.pdf
基于LoRA和 P-Tuning v2 的ChatGLM-6B高效参数微调python源码.zip ChatGLM-6B 是一个清华开源的、支持中英双语的对话语言模型,基于 General Language Model (GLM) 架构,具有 62 亿参数。经过约 1T 标识符的中英...
完整训练代码,对应文章:【AI大模型应用开发】【Fine-Tuning】0. 从一个例子开始学习大模型Fine-Tuning
Apache Kylin - Tuning - Dong Li;Apache Kylin - Tuning - Dong Li
论文“Self-Play Finetuning Converts Weak Language Models to Strong Language Models”的官方实现代码 SPIN 利用自玩机制,允许 LLM 通过与以前的迭代进行博弈来改进自己,而不需要比 SFT 数据集本身更多的人工...
JDK12-hotspot-virtual-machine-garbage-collection-tuning-guide
JDK15-hotspot-virtual-machine-garbage-collection-tuning-guide
JDK11-hotspot-virtual-machine-garbage-collection-tuning-guide
JDK16-hotspot-virtual-machine-garbage-collection-tuning-guide