Skip to content
项目
群组
代码片段
帮助
当前项目
正在载入...
登录 / 注册
切换导航面板
S
seatunnel-web
项目
项目
详情
活动
周期分析
仓库
仓库
文件
提交
分支
标签
贡献者
图表
比较
统计图
议题
0
议题
0
列表
看板
标记
里程碑
合并请求
0
合并请求
0
CI / CD
CI / CD
流水线
作业
日程
统计图
Wiki
Wiki
代码片段
代码片段
成员
成员
折叠边栏
关闭边栏
活动
图像
聊天
创建新问题
作业
提交
问题看板
Open sidebar
宋勇
seatunnel-web
Commits
c7a8f3b2
提交
c7a8f3b2
authored
12月 07, 2023
作者:
李纤
浏览文件
操作
浏览文件
下载
电子邮件补丁
差异文件
修改minio连接
上级
00c4231a
隐藏空白字符变更
内嵌
并排
正在显示
3 个修改的文件
包含
103 行增加
和
5 行删除
+103
-5
pom.xml
...source/seatunnel-datasource-plugins/datasource-s3/pom.xml
+7
-0
S3ClientService.java
...pache/seatunnel/datasource/plugin/s3/S3ClientService.java
+62
-0
S3DatasourceChannel.java
...e/seatunnel/datasource/plugin/s3/S3DatasourceChannel.java
+34
-5
没有找到文件。
seatunnel-datasource/seatunnel-datasource-plugins/datasource-s3/pom.xml
浏览文件 @
c7a8f3b2
...
...
@@ -86,6 +86,13 @@
</exclusions>
</dependency>
<dependency>
<groupId>
io.quarkiverse.minio
</groupId>
<artifactId>
minio-client
</artifactId>
<version>
0.2.0
</version>
</dependency>
<!-- <dependency>-->
<!-- <groupId>org.slf4j</groupId>-->
<!-- <artifactId>slf4j-reload4j</artifactId>-->
...
...
seatunnel-datasource/seatunnel-datasource-plugins/datasource-s3/src/main/java/org/apache/seatunnel/datasource/plugin/s3/S3ClientService.java
0 → 100644
浏览文件 @
c7a8f3b2
package
org
.
apache
.
seatunnel
.
datasource
.
plugin
.
s3
;
import
com.amazonaws.services.dynamodbv2.xspec.S
;
import
io.minio.MinioClient
;
import
io.minio.errors.MinioException
;
public
class
S3ClientService
{
private
String
ENDPOINT
;
private
String
PROVIDER
;
private
String
USERNAME
;
private
String
PASSWORD
;
private
String
BUCKET
;
private
Integer
PORT
;
private
final
String
clientId
=
"Client"
+
(
int
)
(
Math
.
random
()
*
100000000
);
private
MinioClient
minioClient
;
private
String
TOPIC
;
public
S3ClientService
(
String
endpoint
,
String
provider
,
String
username
,
String
password
,
String
bucket
,
Integer
port
)
throws
MinioException
{
this
.
ENDPOINT
=
endpoint
;
this
.
PROVIDER
=
provider
;
this
.
USERNAME
=
username
;
this
.
PASSWORD
=
password
;
this
.
BUCKET
=
bucket
;
this
.
PORT
=
port
;
setMinioClient
(
endpoint
,
provider
,
username
,
password
,
bucket
,
port
);
}
public
MinioClient
getMinioClient
()
{
return
minioClient
;
}
public
void
setMinioClient
(
String
endpoint
,
String
provider
,
String
username
,
String
password
,
String
bucket
,
Integer
port
)
throws
MinioException
{
minioClient
=
new
MinioClient
.
Builder
()
.
endpoint
(
endpoint
,
port
,
true
)
.
credentials
(
username
,
password
)
.
build
();
}
//
// /** 关闭MQTT连接 */
// public void close() throws MqttException {
// mqttClient.close();
// mqttClient.disconnect();
// }
}
seatunnel-datasource/seatunnel-datasource-plugins/datasource-s3/src/main/java/org/apache/seatunnel/datasource/plugin/s3/S3DatasourceChannel.java
浏览文件 @
c7a8f3b2
...
...
@@ -17,7 +17,10 @@
package
org
.
apache
.
seatunnel
.
datasource
.
plugin
.
s3
;
import
io.minio.MinioClient
;
import
org.apache.commons.lang3.StringUtils
;
import
org.apache.seatunnel.api.configuration.util.OptionRule
;
import
org.apache.seatunnel.common.utils.SeaTunnelException
;
import
org.apache.seatunnel.datasource.plugin.api.DataSourceChannel
;
import
org.apache.seatunnel.datasource.plugin.api.DataSourcePluginException
;
import
org.apache.seatunnel.datasource.plugin.api.model.TableField
;
...
...
@@ -69,14 +72,24 @@ public class S3DatasourceChannel implements DataSourceChannel {
@Override
public
boolean
checkDataSourceConnectivity
(
@NonNull
String
pluginName
,
@NonNull
Map
<
String
,
String
>
requestParams
)
{
Configuration
conf
=
HadoopS3AConfiguration
.
getConfiguration
(
requestParams
);
try
(
FileSystem
fs
=
FileSystem
.
get
(
conf
))
{
fs
.
listStatus
(
new
Path
(
"/"
));
if
(
requestParams
.
isEmpty
())
{
throw
new
SeaTunnelException
(
"requestParmas 为空!"
);
}
try
{
S3ClientService
s3Client
=
createS3Client
(
requestParams
);
return
true
;
}
catch
(
IOException
e
)
{
}
catch
(
Exception
ex
)
{
throw
new
DataSourcePluginException
(
String
.
format
(
"check s3 connectivity failed, config is: %s"
,
requestParams
),
e
);
"check s3 connectivity failed, "
+
ex
.
getMessage
(),
ex
);
}
// try (FileSystem fs = FileSystem.get(conf)) {
//// fs.listStatus(new Path("/"));
//
// return true;
// } catch (IOException e) {
// throw new DataSourcePluginException(
// String.format("check s3 connectivity failed, config is: %s", requestParams), e);
// }
}
@Override
...
...
@@ -98,4 +111,20 @@ public class S3DatasourceChannel implements DataSourceChannel {
throw
new
UnsupportedOperationException
(
"getTableFields is not supported for S3 datasource"
);
}
private
S3ClientService
createS3Client
(
Map
<
String
,
String
>
requestParams
)
{
String
endpoint
=
requestParams
.
get
(
"fs.s3a.endpoint"
).
split
(
":"
)[
0
]
+
""
;
Integer
port
=
Integer
.
valueOf
(
requestParams
.
get
(
"fs.s3a.endpoint"
).
split
(
":"
)[
1
]);
String
provider
=
requestParams
.
get
(
"fs.s3a.aws.credentials.provider"
)
+
""
;
String
username
=
requestParams
.
get
(
"access_key"
)
+
""
;
String
password
=
requestParams
.
get
(
"secret_key"
)
+
""
;
String
bucket
=
requestParams
.
get
(
"bucket"
)
+
""
;
try
{
S3ClientService
s3ClientService
=
new
S3ClientService
(
endpoint
,
provider
,
username
,
password
,
bucket
,
port
);
return
s3ClientService
;
}
catch
(
Exception
e
)
{
throw
new
SeaTunnelException
(
"创建Mqtt客户端错误!"
);
}
}
}
编写
预览
Markdown
格式
0%
重试
或
添加新文件
添加附件
取消
您添加了
0
人
到此讨论。请谨慎行事。
请先完成此评论的编辑!
取消
请
注册
或者
登录
后发表评论