Mysql Data to Doris by streamload

导出mysql数据到csv文件

mysql -e "select tracking_number,order_created_time from datamart.t_wide_table_waybill where waybill_submit_time >= '2021-04-01' and waybill_submit_time < '2021-07-01'" > /opt/tmpdata/wide_table_6.csv

tar -zcvf /opt/tmpdata/wide_table_6.csv.tar.gz /opt/tmpdata/wide_table_6.csv

XFTP将压缩文件传输到Doris集群任一台机器A

xftp的传输速度要远快于scp，scp只有700KB/s，XFTP稳定在2.7MB/s，另外传输之前可以先tar压缩，数据可以压缩到原大小的20%以下

在A机器上解压文件

tar -zvf /opt/tmpdata/wide_table_6.csv.tar.gz

Doris streamload方式导入数据时对数据文件的大小限制为10G，若数据文件>10G,则可按10G来切分

split -b 10240m wide_table_6.csv after_split_wide_table_6.csv

用Streamload方式导入数据,直接导入到具体BE节点

curl --location-trusted -u root: -T /data/tmpdata/after_split_wide_table_6.csv -H "label:wide_table_001" -H "max_filter_ratio:1"http://10.63.40.54:8040/api/dwd/dwd_spk_t_wide_table_waybill/_stream_load

由FE节点来分配

curl --location-trusted -u root: -T /data/tmpdata/after_split_wide_table_6.csv -H "label:wide_table_001" -H "max_filter_ratio:1"http://10.63.40.20:8030/api/dwd/dwd_spk_t_wide_table_waybill/_stream_load