153 lines
3.7 KiB
Markdown
153 lines
3.7 KiB
Markdown
|
|
|||
|
|
### 1、新增任务配置说明
|
|||
|
|
|
|||
|
|
a: 任务名称（*必选）
|
|||
|
|
~~~~
|
|||
|
|
任务名称不能超过50个字符 并且 任务名称仅能含数字,字母和下划线
|
|||
|
|
~~~~
|
|||
|
|
|
|||
|
|
b: 运行模式
|
|||
|
|
|
|||
|
|
YARN_PER( yarn独立模式 https://ci.apache.org/projects/flink/flink-docs-release-1.11/zh/ops/deployment/yarn_setup.html#run-a-single-flink-job-on-yarn)
|
|||
|
|
|
|||
|
|
|
|||
|
|
STANDALONE（独立集群 https://ci.apache.org/projects/flink/flink-docs-release-1.11/zh/ops/deployment/cluster_setup.html）
|
|||
|
|
|
|||
|
|
|
|||
|
|
LOCAL(本地集群 https://ci.apache.org/projects/flink/flink-docs-release-1.11/zh/ops/deployment/local.html )
|
|||
|
|
|
|||
|
|
|
|||
|
|
<font color=red size=5>LOCAL 需要在本地单机启动flink 服务 ./bin/start-cluster.sh </font>
|
|||
|
|
|
|||
|
|
|
|||
|
|
c: flink运行配置
|
|||
|
|
|
|||
|
|
<font color=red size=5>1、YARN_PER模式 </font>
|
|||
|
|
|
|||
|
|
~~~~
|
|||
|
|
|
|||
|
|
参数（和官方保持一致）但是只支持 -yD -p -yjm -yn -ytm -ys -yqu(必选)
|
|||
|
|
-ys slot个数。
|
|||
|
|
-yn task manager 数量。
|
|||
|
|
-yjm job manager 的堆内存大小。
|
|||
|
|
-ytm task manager 的堆内存大小。
|
|||
|
|
-yqu yarn队列明
|
|||
|
|
-p 并行度
|
|||
|
|
-yD 如-yD taskmanager.heap.mb=518
|
|||
|
|
详见官方文档
|
|||
|
|
如： -yqu flink -yjm 1024m -ytm 2048m -p 1 -ys 1
|
|||
|
|
|
|||
|
|
~~~~
|
|||
|
|
|
|||
|
|
<font color=red size=5>2、LOCAL模式 </font>
|
|||
|
|
~~~~
|
|||
|
|
无需配置
|
|||
|
|
~~~~
|
|||
|
|
|
|||
|
|
|
|||
|
|
<font color=red size=5>3、STANDALONE模式 </font>
|
|||
|
|
~~~~
|
|||
|
|
-d,--detached If present, runs the job in detached
|
|||
|
|
mode
|
|||
|
|
|
|||
|
|
-p,--parallelism <parallelism> The parallelism with which to run the
|
|||
|
|
program. Optional flag to override the
|
|||
|
|
default value specified in the
|
|||
|
|
configuration.
|
|||
|
|
|
|||
|
|
-s,--fromSavepoint <savepointPath> Path to a savepoint to restore the job
|
|||
|
|
from (for example
|
|||
|
|
hdfs:///flink/savepoint-1537).
|
|||
|
|
|
|||
|
|
其他运行参数可通过 flink -h查看
|
|||
|
|
~~~~
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
d: 三方地址
|
|||
|
|
~~~~
|
|||
|
|
填写连接器或者udf等jar
|
|||
|
|
如：
|
|||
|
|
http://ccblog.cn/jars/flink-connector-jdbc_2.11-1.12.0.jar
|
|||
|
|
http://ccblog.cn/jars/flink-sql-connector-kafka_2.11-1.12.0.jar
|
|||
|
|
http://ccblog.cn/jars/flink-streaming-udf.jar
|
|||
|
|
http://ccblog.cn/jars/mysql-connector-java-5.1.25.jar
|
|||
|
|
|
|||
|
|
地址填写后 udf可以在sql语句里面直接写
|
|||
|
|
CREATE FUNCTION jsonHasKey as 'com.xx.udf.JsonHasKeyUDF';
|
|||
|
|
~~~~
|
|||
|
|
|
|||
|
|
|
|||
|
|
多个url使用换行
|
|||
|
|
### 或者： 如果使用三方jar管理功能 可直接填写jar包名字
|
|||
|
|
|
|||
|
|
flink-connector-jdbc_2.11-1.12.0.jar
|
|||
|
|
|
|||
|
|
flink-streaming-udf.jar
|
|||
|
|
|
|||
|
|
mysql-connector-java-5.1.25.jar
|
|||
|
|
|
|||
|
|
udf 开发demo 详见 [https://github.com/zhp8341/flink-streaming-udf](https://github.com/zhp8341/flink-streaming-udf)
|
|||
|
|
|
|||
|
|
|
|||
|
|
e: sql语句
|
|||
|
|
#### 提前先在hive test的库下创建好test的表
|
|||
|
|
~~~~
|
|||
|
|
create table test(
|
|||
|
|
id int,
|
|||
|
|
name string
|
|||
|
|
)
|
|||
|
|
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t';
|
|||
|
|
~~~~
|
|||
|
|
|
|||
|
|
|
|||
|
|
~~~~sql
|
|||
|
|
CREATE CATALOG testmyhive WITH (
|
|||
|
|
'type' = 'hive',
|
|||
|
|
'default-database' = 'test',
|
|||
|
|
'hive-conf-dir' = '/alidata/server/zhp/catalog/config'
|
|||
|
|
);
|
|||
|
|
USE CATALOG testmyhive;
|
|||
|
|
|
|||
|
|
insert into test.test values(4,'n2');
|
|||
|
|
|
|||
|
|
~~~~
|
|||
|
|
|
|||
|
|
|
|||
|
|
### 2、系统设置
|
|||
|
|
|
|||
|
|
~~~~
|
|||
|
|
|
|||
|
|
系统设置有三个必选项
|
|||
|
|
1、flink-streaming-platform-web应用安装的目录（必选）
|
|||
|
|
这个是应用的安装目录
|
|||
|
|
如 /root/flink-streaming-platform-web/
|
|||
|
|
|
|||
|
|
2、flink安装目录（必选）
|
|||
|
|
--flink客户端的目录 如： /usr/local/flink-1.12.0/
|
|||
|
|
|
|||
|
|
3、yarn的rm Http地址
|
|||
|
|
--hadoop yarn的rm Http地址 http://hadoop003:8088/
|
|||
|
|
|
|||
|
|
4、flink_rest_http_address
|
|||
|
|
LOCAL模式使用 flink http的地址
|
|||
|
|
|
|||
|
|
5、flink_rest_ha_http_address
|
|||
|
|
STANDALONE模式 支持HA的 可以填写多个地址 ;用分隔
|
|||
|
|
|
|||
|
|
~~~~
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|
|||
|
|
|