H5下的图片上传下载与dataURI转换

最近加强了一点点对前端的学习,从各个地方查找和学习到了前端图片基础处理的一些知识记录一下。

先看需要解决的问题:

  1. 粘贴与拖拽上传: 这部分我阅读了一下iview的上传的组件的源码,基本上就是监听元素的触发事件的关联对象: 拖拽
1
2
3
const onDrop = function (e) {
  this.uploadFiles(e.dataTransfer.files);
};

粘贴:

1
2
3
const handlePaste = function (e) {
  this.uploadFiles(e.clipboardData.files);
};

本质上是通过事件e触发后读取关联的对象(拖拽行为的dataTransfer和粘贴行为的clipboardData)下属的files数组(内容为File对象的数组列表)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
const uploadFiles = function (files) {
    let postFiles = Array.prototype.slice.call(files);
    if (!this.multiple) postFiles = postFiles.slice(0, 1);

    if (postFiles.length === 0) return;

    postFiles.forEach(file => {
        this.upload(file);
    });
};

uploadFiles遍历files数组后逐个调用上传接口

  1. 上传图片的预览

现在的chrome的安全机制已经不能允许直接使用input file的源路径直接展示内容了。 不过我们还是可以通过File Reader的方式将需要读取的图片文件转为dataURI的形式并赋值给img的src直接呈现出来。

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
// 创建一个 FileReader 对象
let reader = new FileReader();
// readAsDataURL 方法用于读取指定 Blob 或 File 的内容
// 当读操作完成,readyState 变为 DONE,loadend 被触发,此时 result 属性包含数据:URL(以 base64 编码的字符串表示文件的数据)
// 读取文件作为 URL 可访问地址
reader.readAsDataURL(file);

const _this = this;
// eslint-disable-next-line no-unused-vars
reader.onloadend = function(e) {
_this.handlePreview(reader.result);
};

其中的reader.result就是文件内容的base64编码后的dataURI地址链接直接赋值给img的src即可呈现出来

  1. 远程图片的下载

这个直接在网上找到了处理的方法,记录一下代码:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
const getBase64ImageFromUrl = async function(imageUrl) {
    let res = await fetch(imageUrl);
    let blob = await res.blob();
    return new Promise((resolve, reject) => {
      let reader = new FileReader();
      reader.addEventListener(
        "load",
        function() {
          resolve(reader.result);
        },
        false
      );
      reader.onerror = () => {
        return reject(this);
      };
      reader.readAsDataURL(blob);
    });
};

本质上还是利用FileReader转换fetch下载回来的blob内容转换为dataURI的链接

  1. dataURI的上传

要做到dataURI的上传首先先了解一下思路:

  • 将dataURI转化为File对象
  • 将File对象和其他的提交字段一并添加到FormData的对象实例
  • 将FormData通过axios或者原生等ajax方式转化为和普通上传文件表单一样的http报文的形式实现提交上传请求

基础的就是将dataURI的字符串拆解并转换回Blob对象:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
const dataURItoBlob = function(dataURI) {
  // convert base64 to raw binary data held in a string

  // doesn't handle URLEncoded DataURIs - see SO answer #6850276 for code that does this
  let byteString;
  if (dataURI.split(",")[0].indexOf("base64") >= 0) {
    byteString = atob(dataURI.split(",")[1]);
  } else {
    byteString = unescape(dataURI.split(",")[1]);
  }

  // separate out the mime component

  let mimeString = dataURI
    .split(",")[0]
    .split(":")[1]
    .split(";")[0];

  // write the bytes of the string to an ArrayBuffer

  let ab = new ArrayBuffer(byteString.length);

  // create a view into the buffer

  let ia = new Uint8Array(ab);

  // set the bytes of the buffer to the correct values

  for (var i = 0; i < byteString.length; i++) {
    ia[i] = byteString.charCodeAt(i);
  }

  // write the ArrayBuffer to a blob, and you're done

  let blob = new Blob([ab], { type: mimeString });
  return blob;
};

深入去看一下File对象的接口会发现其实File对象就是Blob对象的装饰器,其实就是一个壳子加上了name,lastmodified等属性

所以简单改造一下:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
dataURItoFile = function(dataURI, filename, lastModified) {
  // convert base64 to raw binary data held in a string

  // doesn't handle URLEncoded DataURIs - see SO answer #6850276 for code that does this
  let byteString;
  if (dataURI.split(",")[0].indexOf("base64") >= 0) {
    byteString = atob(dataURI.split(",")[1]);
  } else {
    byteString = unescape(dataURI.split(",")[1]);
  }

  // separate out the mime component

  let mimeString = dataURI
    .split(",")[0]
    .split(":")[1]
    .split(";")[0];
  if (!lastModified) {
    lastModified = Date.now();
  }
  if (!filename) {
    let ext = mimeString.split("/")[1] || "bin";
    filename = `upload.${ext}`;
  }
  // write the bytes of the string to an ArrayBuffer

  let ab = new ArrayBuffer(byteString.length);

  // create a view into the buffer

  let ia = new Uint8Array(ab);

  // set the bytes of the buffer to the correct values

  for (var i = 0; i < byteString.length; i++) {
    ia[i] = byteString.charCodeAt(i);
  }

  // write the ArrayBuffer to a blob, and you're done

  let file = new File([ab], filename, {
    type: mimeString,
    lastModified: lastModified
  });
  return file;
};

再加上点小辅助函数:

1
2
3
const checkIsDataURI = function(dataURI) {
  return dataURI.split(",")[0].indexOf("base64") >= 0;
};

得到File对象以后可以和其他需要一并提交的数据组成FormData对象实例:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
let srcDataURI = 'data:image/png;base64,xxx此处省略base64编码后的内容';
let dataFile = dataURItoFile(srcDataURI);
let fileFieldName = 'upload_file';
let addonData = {
  field1:'val1',
  field2:'val2',
};
let postData = new FormData();
  postData.append(fileFieldName, dataFile, dataFile.name);
  if (addonData) {
    Object.keys(addonData).forEach(function(key) {
      postData.append(key, addonData[key]);
    });
  }

然后再整体封装成普通的表单上传文件的头+内容的方式,此处使用axios处理:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
const postUploadDataUri = function(uploadUrl, dataURI, addonData, filename, field, authToken) {
      let dataFile = dataURItoFile(dataURI, filename);
      let postData = new FormData();
      if (!field) {
        field = "upload_file";
      }
      postData.append(field, dataFile, dataFile.name);
      if (addonData) {
        Object.keys(addonData).forEach(function(key) {
          postData.append(key, addonData[key]);
        });
      }
      let requestConfig = {
        headers: {
          Authorization: `Bearer ${authToken}`
        }
      };
      this.$axios
        .post(uploadUrl, postData, requestConfig)
        .then(function(response) {
          console.log(response);
        })
        .catch(function(error) {
          console.log(error);
        });
    }

注意由于post的参数对象是FormData对象类型,headers的配置中不需要也不能傻乎乎的加上:

1
{ "Content-Type": "multipart/form-data" }

虽然文件上传的确需要multipart/form-data的头,但完整的报文中form-data的头里,各个文件和附加字段之间是会有boundary的分割标记字符串来告知服务端上传的各个数据流的内容到了服务端那边如何解析报文的 类似于这样:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
POST http://www.example.com HTTP/1.1
Content-Type:multipart/form-data; boundary=----WebKitFormBoundaryFw0bBOkuhsMWbwBV
 
------WebKitFormBoundaryFw0bBOkuhsMWbwBV
Content-Disposition: form-data; name="field1"
 
val1
 
------WebKitFormBoundaryFw0bBOkuhsMWbwBV
Content-Disposition: form-data; name="field2"
 
val2
------WebKitFormBoundaryFw0bBOkuhsMWbwBV
Content-Disposition: form-data; name="upload_file"; filename="example.png"
Content-Type: image/png
 
PNG ... content of example.png ...

boundary的分割标记随机字符串axios封装的XMLHttpRequest对象会自动和Content-Type:multipart/form-data一并帮我们生成并加上所以并不用自己手工处理这块的头信息

至此基本上可以解决大部分dataURI的上传处理问题

mysql数据库空间的influxDB增量统计

业务系统中RDS的mysql的空间捉襟见肘时就会想起来需要做一下容量的统计和趋势监控 实施的方案是:

定时用python从mysql的information_schema库中的TABLES表中抽取DATA_LENGTH+INDEX_LENGTH的数据以及DATA_FREE的数据来做统计项 将数据抽取到influxDB中备查

核心的查询语句:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
        # 1048576 字节 等于 1MB
        # 监控门槛:
        # DATA_FREE>= 6MB (可优化空间大于6MB)
        # TABLE_ROWS>= 100k (10万行以上)
        # DATA_LENGTH>= 20MB (数据空间占用大于20MB)
        # INDEX_LENGTH>= 20MB (索引空间占用大于20MB)
        # `TABLE_SCHEMA` IN ('{"','".join(self.check_dbs)}')
        sql = f""" SELECT `TABLE_SCHEMA`,`TABLE_NAME`, CONCAT_WS('.',`TABLE_SCHEMA`,`TABLE_NAME`) AS DBTABLENAME,
(`DATA_FREE`/1048576) AS free_len_mb,
(`DATA_LENGTH`+ `INDEX_LENGTH`)/1048576 AS row_len_mb,
(`TABLE_ROWS`/1000) AS rows_kb ,
(`DATA_LENGTH`/1048576) AS dat_len_mb,
(`INDEX_LENGTH`/1048576) AS idx_len_mb
FROM `information_schema`.`TABLES` WHERE
`TABLE_SCHEMA` IN ('{"','".join(self.check_dbs)}')
AND (`DATA_FREE`>=6291456 OR TABLE_ROWS >= 100000 OR DATA_LENGTH>=20971520 OR INDEX_LENGTH>=15728640 )
ORDER BY free_len_mb DESC,row_len_mb DESC,dat_len_mb DESC,idx_len_mb DESC, `TABLE_SCHEMA` ASC,`TABLE_NAME` ASC
"""

然后单行的influxdb的表结构:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
    tsdb_item_record = {
        'measurement': 'dbtable_stats',
        'tags': {
            'db': db,
            'table':table,
            'dbtable':dbtable,
        },
        'time': f'{utcnow.strftime("%Y-%m-%dT%H:%M:%S")}Z',
        'fields': {
            'free_len_mb':free_len_mb,
            'row_len_mb':row_len_mb,
            'rows':rows_kb,
            'dat_len_mb':dat_len_mb,
            'idx_len_mb':idx_len_mb
        }
    }

这样在influxdb中我们就有了每个埋点间隔下统计的数据信息,接下来在grafana中可以用InfluxDB的查询函数统计出对应的信息并展示图表

按时间查询各个表的空间占用(数据+索引)的增长量:

1
SELECT cumulative_sum(difference(mean("row_len_mb"))) FROM "dbspace_watches_rp"."dbtable_stats" WHERE $timeFilter GROUP BY time(5m), "dbtable" fill(null)

再加上一下查询起始时的值就是当下的总数据量的表数据空间排名:

1
SELECT first("row_len_mb") + cumulative_sum(difference(mean("row_len_mb"))) FROM "dbspace_watches_rp"."dbtable_stats" WHERE $timeFilter GROUP BY time(5m), "dbtable" fill(null)

image-20200826173247724

同理也可以得到库表的DATA_FREE也就是可优化空间的增长量和排名在此不再赘述 另外提一句,mysql在做optmize释放空间的时候会锁表,所以还是在业务空闲期做这些释放空间的活比较好

另外可以顺便求一下倒数得到查询时间区间内的各表的增长率,方便发现特别高增长的表做对应的优化

1
SELECT derivative(cumulative_sum(difference(mean("row_len_mb"))), 10m) FROM "dbspace_watches_rp"."dbtable_stats" WHERE $timeFilter GROUP BY time(1h), "dbtable" fill(null)

OpenCv开发中所遇到的问题备忘

1.qt库冲突的问题:

1
2
3
You might be loading two sets of Qt binaries into the same process. Check that all plugins are compiled against the right Qt binaries. Export DYLD_PRINT_LIBRARIES=1 and check that only one set of binaries are being loaded.
QObject::moveToThread: Current thread (0x7ff56d74feb0) is not the object's thread (0x7ff570dfec60).
Cannot move to target thread (0x7ff56d74feb0)

运行conda环境下opencv时遇到的,查了一下大概是需要卸载了opencv-python后再安装一下opencv-python-headless的版本

1
2
pip uninstall opencv-python
pip install opencv-python-headless

安装时长不够就用:

1
pip install --default-timeout=50000  opencv-python-headless

参考: https://www.pythonheidong.com/blog/article/410902/