s3 api接口的调用

最近公司使用s3做文件存储服务器,因此在程序中需要调用s3的api,目前程序中使用了python和java版本的s3的api,简单做下记录,方便以后使用。

一、s3 api使用python版

1.安装boto3的sdk

pip install boto3==1.4.7

2.创建s3的api客户端

import boto3
from botocore.client import Config
import os
import sys
import threading

#创建s3链接,如果s3服务器是第四代,则需要设置signature_version=‘s3v4‘
s3_client = boto3.client(‘s3‘, endpoint_url=‘s3服务器地址‘,
                         aws_access_key_id=‘s3服务器的access_key_id‘,
                         aws_secret_access_key=‘s3服务器的secret_access_key‘,
                         region_name=‘s3服务器的时区,这里可以填写cn-north-1‘,
                         config=Config(signature_version=‘s3‘))

3.获取s3中bucket列表

bucket_list = s3_client.list_buckets()

print bucket_list

4.创建bucket

bucket = s3_client.create_bucket(Bucket=‘bucket的名称‘)

print bucket

5.获取bucket信息
bucket_info = s3_client.head_bucket(Bucket=‘bucket的名称‘)

print bucket_info

6.删除bucket
#bucket_delete = s3_client.delete_bucket(Bucket=‘bucket的名称‘)

7.上传文件到s3服务器

# s3_client.upload_file("上传的源文件地址", "bucket名称", "上传文件在s3上对应的key名称", ExtraArgs={‘ACL‘: ‘public-read‘})

或者

s3_client.upload_file("上传的源文件地址", "bucket名称", "上传文件在s3上对应的key名称",

      ExtraArgs={‘ACL‘: ‘public-read‘},

      Callback=UploadProgressPercentage("上传的源文件地址"))
Callback属性对应的类方法如下,该类方法在控制台中打印了上传文件的进度

class UploadProgressPercentage(object):
    def __init__(self, filename):
        self._filename = filename
        self._size = float(os.path.getsize(filename))
        self._seen_so_far = 0
        self._lock = threading.Lock()
    def __call__(self, bytes_amount):
        # To simplify we‘ll assume this is hooked up
        # to a single filename.
        with self._lock:
            self._seen_so_far += bytes_amount
            percentage = (self._seen_so_far / self._size) * 100
            sys.stdout.write(
                "\r%s  %s / %s  (%.2f%%)" % (
                    self._filename, self._seen_so_far, self._size,
                    percentage))
            sys.stdout.flush()

8.获取bucket下文件列表

object_list = s3_client.list_objects(Bucket=‘bucket名称‘)
print object_list

9.查看bucket下的某个文件信息
object_info = s3_client.get_object(Bucket=‘bucket名称‘, Key=‘文件对应的key名称‘)
print object_info
10.删除文件
object_delete = s3_client.delete_object(Bucket=‘bucket名称‘, Key=‘文件对应的key名称‘)

11.下载文件

s3_client.download_file("bucket名称", "文件对应的key名称", "文件下载到的地址")

或者

s3client.download_file("bucket名称", "文件对应的key名称", "文件下载到的地址",Callback=DownloadProgressPercentage("文件下载到的地址"))

Callback属性对应的类方法如下,该类方法在控制台中打印了下载文件的进度

class _DownloadProgressPercentage(object):
    def __init__(self, filename):
        self._filename = filename
        self._seen_so_far = 0
        self._lock = threading.Lock()
    def __call__(self, bytes_amount):
        # To simplify we‘ll assume this is hooked up
        # to a single filename.
        with self._lock:
            self._seen_so_far += bytes_amount
            sys.stdout.write(
                "\r%s --> %s bytes transferred" % (
                    self._filename, self._seen_so_far))
            sys.stdout.flush()

参考文档

boto3 api官方文档:http://boto3.readthedocs.io/en/latest/guide/quickstart.html

1.s3的client的api

2.s3的client的api上传下载示例

二、3 api使用java版

1.在pom.xml中增加依赖包

<dependency>
  <groupId>com.amazonaws</groupId>
  <artifactId>aws-java-sdk</artifactId>
  <version>1.7.3</version>
</dependency>

2.java中使用s3的api的demo

package org.jenkinsci.plugins.s3_step;

import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.URL;
import java.security.SecureRandom;
import java.util.ArrayList;
import java.util.List;

import com.amazonaws.ClientConfiguration;
import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.regions.Region;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.AbortMultipartUploadRequest;
import com.amazonaws.services.s3.model.Bucket;
import com.amazonaws.services.s3.model.BucketPolicy;
import com.amazonaws.services.s3.model.CannedAccessControlList;
import com.amazonaws.services.s3.model.CompleteMultipartUploadRequest;
import com.amazonaws.services.s3.model.CompleteMultipartUploadResult;
import com.amazonaws.services.s3.model.CopyObjectResult;
import com.amazonaws.services.s3.model.GeneratePresignedUrlRequest;
import com.amazonaws.services.s3.model.GetBucketPolicyRequest;
import com.amazonaws.services.s3.model.GetObjectRequest;
import com.amazonaws.services.s3.model.InitiateMultipartUploadRequest;
import com.amazonaws.services.s3.model.InitiateMultipartUploadResult;
import com.amazonaws.services.s3.model.ListMultipartUploadsRequest;
import com.amazonaws.services.s3.model.ListObjectsRequest;
import com.amazonaws.services.s3.model.ListPartsRequest;
import com.amazonaws.services.s3.model.MultipartUpload;
import com.amazonaws.services.s3.model.MultipartUploadListing;
import com.amazonaws.services.s3.model.ObjectListing;
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.amazonaws.services.s3.model.PartETag;
import com.amazonaws.services.s3.model.PartListing;
import com.amazonaws.services.s3.model.PartSummary;
import com.amazonaws.services.s3.model.PutObjectRequest;
import com.amazonaws.services.s3.model.PutObjectResult;
import com.amazonaws.services.s3.model.S3Object;
import com.amazonaws.services.s3.model.S3ObjectInputStream;
import com.amazonaws.services.s3.model.S3ObjectSummary;
import com.amazonaws.services.s3.model.UploadPartRequest;
import com.amazonaws.services.s3.model.UploadPartResult;

public class Demo {
    AWSCredentials credentials = null;
    public AmazonS3 getS3Client(String access_key, String secret_key,
                                String endpoint) {
        ClientConfiguration conf = null;
        if (conf == null) {
            conf = new ClientConfiguration();
            credentials = new BasicAWSCredentials(access_key, secret_key);
        }
        AmazonS3 s3 = new AmazonS3Client(credentials, conf);
        s3.setRegion(Region.getRegion(Regions.CN_NORTH_1));
        if (endpoint != null)
            s3.setEndpoint(endpoint);
        return s3;
    }

// ============================ bucket ================================
    public void testCreateBucket(AmazonS3 s3Client, String bucketName) {
        System.out.println("======================创建 Bucket==========================");
        Bucket bucket = s3Client.createBucket(bucketName);
        System.out.println(bucket);
        System.out.println("创建 Bucket : " + bucket.getName());
        System.out.println("======================创建 Bucket==========================");
    }

public void testDeleteBucket(AmazonS3 s3Client, String bucketName) {
        System.out.println("======================删除 Bucket==========================");
        System.out.println("删除 Bucket : " + bucketName);
        s3Client.deleteBucket(bucketName);
        System.out.println("======================删除 Bucket==========================");
    }

public void testDoesBucketExist(AmazonS3 s3Client, String bucketName) {
        System.out.println("======================Bucket 是否存在==========================");
        boolean bucketExist = s3Client.doesBucketExist(bucketName);
        System.out.println(String.format("%s : %s", bucketName, bucketExist));
        System.out.println("======================Bucket 是否存在==========================");
    }

public void testListBuckets(AmazonS3 s3Client) {
        System.out.println("======================枚举 Buckets==========================");
        List<Bucket> buckets = s3Client.listBuckets();
        for (Bucket bucket : buckets) {
            System.out.println(String.format("%s - %s - %s", bucket.getName(), bucket.getOwner(), bucket.getCreationDate()));
        }
        System.out.println("======================枚举 Buckets==========================");
    }

public void testListObjects(AmazonS3 s3Client, String bucketName) {
        System.out.println("======================枚举 Objects==========================");
        List<String> objectKeys = new ArrayList<String>();
        String prefix = "/s3demo/test/";
//        for (int i = 0; i < 5; i++) {
//            objectKeys.add(prefix + i);
//        }
//        for (String objectKey : objectKeys) {
//            testPutObject(s3Client, bucketName, objectKey);
//        }

ObjectListing objects = s3Client.listObjects(bucketName);
        List<S3ObjectSummary> s3Objects = objects.getObjectSummaries();
        for (S3ObjectSummary s3os : s3Objects) {
            System.out.println(String.format("%s - %s - %s", s3os.getKey(), s3os.getETag(), s3os.getSize()));
        }

// prefix
        ListObjectsRequest listObjReq = new ListObjectsRequest();
        listObjReq.setBucketName(bucketName);
        listObjReq.setDelimiter("/");
        listObjReq.setPrefix(prefix);

ObjectListing preobjects = s3Client.listObjects(listObjReq);
        List<String> preobjs = preobjects.getCommonPrefixes();
        for (String preobj : preobjs) {
            System.out.println(preobj + "\n");
        }
        System.out.println("======================枚举 Objects==========================");
    }

// ============================ bucket ================================

// ============================ object ================================
    public void testPutObject(AmazonS3 s3Client, String bucketName, String objectKey,String file_source_address) {
      System.out.println("======================上传 object==========================");
        s3Client.putObject(new PutObjectRequest(bucketName, objectKey, new File(file_source_address)).withCannedAcl(CannedAccessControlList.PublicRead));
        System.out.println("======================上传 object==========================");
    }

public void testGetObject(AmazonS3 s3Client, String bucketName,String objectKey) throws Exception {
        System.out.println("======================下载 object==========================");
        try {
            S3Object s3Object = s3Client.getObject(new GetObjectRequest(bucketName,objectKey));
            //下载文件到指定地方
            InputStream reader = new BufferedInputStream(s3Object.getObjectContent());
            File file = new File("localFilename");    
            FileOutputStream fos = new FileOutputStream(file);
            OutputStream writer = new BufferedOutputStream(fos);
            int read = -1;
            while ( ( read = reader.read() ) != -1 ) {
                        writer.write(read);
            }
            writer.flush();
            writer.close();
            reader.close();
            //获取一个request,生成一个下载的url地址,可以在浏览器上下载
            GeneratePresignedUrlRequest urlRequest = new GeneratePresignedUrlRequest(bucketName, objectKey);  
            //生成公用的url    
            URL url = s3Client.generatePresignedUrl(urlRequest);    
            System.out.println("=========URL=================" + url + "============URL=============");    
            if (url == null) {    
                throw new Exception("can‘t get s3 file url!");    
            }    
            System.out.println("URL=" + url.toString());    
        } catch (Exception e) {
            e.printStackTrace();
        }
        
        System.out.println("======================下载 object==========================");
    }

public void testDeleteObject(AmazonS3 s3Client, String bucketName,String objectKey) {
        System.out.println("======================删除 object==========================");
        s3Client.deleteObject(bucketName, objectKey);
        System.out.println("======================删除 object==========================");
    }

public void testCopyObject(AmazonS3 s3Client, String srcBucketName, String srcObjectKey, String dstBucketName, String dstObjectKey) {
        System.out.println("======================复制 object==========================");
        CopyObjectResult cpres = s3Client.copyObject(srcBucketName, srcObjectKey, dstBucketName, dstObjectKey);
        String text = String.format("test copy objcet %s from %s to %s.%s", srcObjectKey, srcBucketName, dstBucketName, dstObjectKey);
        System.out.println(text);
        System.out.println("object etag : " + cpres.getETag());
        System.out.println("======================复制 objects==========================");
    }

public void testMultiUpload(AmazonS3 s3Client, String bucketName, String objectKey) {
        System.out.println("======================分块上传==========================");
        System.out.println("初始化分块上传");
        InitiateMultipartUploadRequest initMultiUp = new InitiateMultipartUploadRequest(bucketName, objectKey);
        InitiateMultipartUploadResult initMultiUpRes = s3Client.initiateMultipartUpload(initMultiUp);
        String uploadId = initMultiUpRes.getUploadId();
        System.out.println("分块上传 MultiUpload Id : " + uploadId);

System.out.println("上传块");
        List<PartETag> partETags = new ArrayList<PartETag>();
        for (int i = 1; i <= 3; i++) {
            UploadPartRequest uploadPartReq = new UploadPartRequest();
            uploadPartReq.setBucketName(bucketName);
            uploadPartReq.setKey(objectKey);
            uploadPartReq.setUploadId(uploadId);
            uploadPartReq.setPartNumber(i);
            byte[] part = new byte[1024 * 1024 * 6]; // 分块上传一个块必须大于等于5M
            if (i == 3) {
                uploadPartReq.setLastPart(true);
                part = new byte[1024 * 1024 * 3]; // 最后一个块可以小于5M
            }
            SecureRandom random = new SecureRandom();
            random.nextBytes(part);
            System.out.println("块大小 = " + part.length);
            uploadPartReq.setPartSize(part.length);
            InputStream inputStream = new ByteArrayInputStream(part);
            uploadPartReq.setInputStream(inputStream);
            UploadPartResult uploadPartRes = s3Client.uploadPart(uploadPartReq);
            PartETag partETag = uploadPartRes.getPartETag();
            partETags.add(partETag);
            System.out.println(String.format("上传块 %s - %s", partETag.getPartNumber(), partETag.getETag()));
        }

CompleteMultipartUploadRequest comMultiReq = new CompleteMultipartUploadRequest(
                bucketName, objectKey, uploadId, partETags);
        CompleteMultipartUploadResult comMultires = s3Client.completeMultipartUpload(comMultiReq);
        System.out.println("完成上传 etag : " + comMultires.getETag());

System.out.println("测试终止上传");
        String objectKey1 = "test_abort_multi_upload";
        InitiateMultipartUploadRequest initMultiUp1 = new InitiateMultipartUploadRequest(
                bucketName, objectKey1);
        InitiateMultipartUploadResult initMultiUpRes1 = s3Client
                .initiateMultipartUpload(initMultiUp1);
        String uploadId1 = initMultiUpRes1.getUploadId();
        System.out.println("终止上传 upload id  : " + uploadId1);
        AbortMultipartUploadRequest abortMultiUp = new AbortMultipartUploadRequest(bucketName, objectKey1, uploadId1);
        s3Client.abortMultipartUpload(abortMultiUp);
        System.out.println("======================分块上传==========================");
    }

public void testListUploadIds(AmazonS3 s3Client, String bucketName) {
        System.out.println("======================枚举分块上传uploadId==========================");
        String keyPrefix = "/demo/listupids";
        for (int i = 1; i < 5; i++) {
            String objectKey = keyPrefix + i;
            InitiateMultipartUploadRequest initMultiUp = new InitiateMultipartUploadRequest(bucketName, objectKey);
            s3Client.initiateMultipartUpload(initMultiUp);
        }

ListMultipartUploadsRequest listMultiUpReq = new ListMultipartUploadsRequest(bucketName);
        MultipartUploadListing multiUploads = s3Client.listMultipartUploads(listMultiUpReq);
        List<MultipartUpload> uploads = multiUploads.getMultipartUploads();
        for (MultipartUpload up : uploads) {
            System.out.println(String.format("uploadId : %s - key : %s", up.getUploadId(), up.getKey()));
        }
        System.out.println("======================枚举分块上传uploadId==========================");
    }

public void testListMultiParts(AmazonS3 s3Client, String bucketName, String objectkey, String uploadId) {
        System.out.println("======================枚举分块上传part==========================");
        ListPartsRequest listPartsReq = new ListPartsRequest(bucketName, objectkey, uploadId);
        PartListing parts = s3Client.listParts(listPartsReq);
        List<PartSummary> ps = parts.getParts();
        for (PartSummary p : ps) {
            System.out.println(String.format("partNum : %s - partEtag : %s", p.getPartNumber(), p.getETag()));
        }
        System.out.println("======================枚举分块上传part==========================");
    }
    
    public void testIsFileExists(AmazonS3 s3Client, String s3Path,String keyName){
        System.out.println("======================判断文件是否已经存在==========================");
        try{
            boolean flag = false;
            S3Object s3Object = s3Client.getObject(s3Path, keyName);
            if(s3Object!=null){
                flag = true;
            }
            if(flag){
                System.out.println("File exists,filename:"+keyName);
            }
        }catch(Exception e){
            e.printStackTrace();
        }
    }
    // ============================ object ================================

public static void main(String[] args) throws IOException {
        String access_key = "s3的access key";
        String secret_key = "s3的secret key";
        String endpoint = "s3的地址";

Demo d = new Demo();
        AmazonS3 s3Client = d.getS3Client(access_key, secret_key, endpoint);
        String bucketName = "demo";
        String bucketName1 = "demo1";
        //创建bucket
//        d.testCreateBucket(s3Client, bucketName);
        //查询bucket列表
//        d.testListBuckets(s3Client);
        //判断bucket是否存在
//        d.testDoesBucketExist(s3Client, bucketName);
        //删除bucket
//        d.testDeleteBucket(s3Client, bucketName2);
        //上传文件
//        String objectKey = "var/lib/pom.xml";
//        String objectKey1 = "var/lib/pom.xml";
//        String file_source_address = "E:/demo/test1/pom.xml";
//        d.testPutObject(s3Client, bucketName, objectKey,file_source_address);
        //下载文件
//        d.testGetObject(s3Client, bucketName, objectKey);
//        //复制文件
//        d.testCopyObject(s3Client, bucketName, objectKey, bucketName1, objectKey1);
        //bucket下面文件列表
//        d.testListObjects(s3Client, bucketName);
        //判断文件是否存在
//        d.testIsFileExists(s3Client, bucketName, objectKey);

}
}

时间: 2024-11-08 03:20:05

s3 api接口的调用的相关文章

Saltstack的API接口与调用方式

saltstack看起来是成为一个大规模自己主动化运维和云计算管理的一个框架,类似于SDK,并非像puppet仅仅成为一个工具.基于良好设计的API和清楚的思路,让salt的二次开发变得非常easy.写非常少的代码就能够将salt跟现有的运维系统结合.saltstack是用python语言实现的,假设对saltstack本身进行二次开发,就必须得会python语言. Saltstack是通过多个独立的模块构成的,这些都能够当做saltstack的api.然后在上层做基础开发能够通过调用这些sal

Web车牌识别中API接口的调用

Web Service车牌识别就是服务器版车牌OCR识别软件,属于后端识别,Web Service车牌识别软件目前支持Windows.Linux等主流服务器操作系统. 该软件可部署在私有服务器中(私有本地服务器或云服务器均可),APP和业务系统可通过web service接口调用该识别服务,设备端只负责拍摄图像后上传,上传到已部署Web Service车牌识别软件的服务器中进行识别,识别完成后再返回标准的XML数据.整个识别过程和我公司没有任何交互,均在企业自有的服务器上完成识别. 服务主要功能

电子面单api接口_JAVA调用示例

电子面单是一种通过热敏纸打印输出纸质物流面单的物流服务.通过热感应显示文字,打印速度比传统针式打印速度提升4~6倍.电子面单以接口形式嵌入到自己的系统.网站上,可以在自己的平台操作打印电子面单. 目前电子面单接口类型有:快递公司自行提供接口.菜鸟电子面单接口.快递鸟电子面单接口.菜鸟电子面单接口.快递鸟电子面单接口接入方式类似,都支持15家快递查询,其中菜鸟不支持顺丰电子面单打印,快递鸟不支持天天.快捷等快递电子面单打印.以下先做快递鸟电子面单接口的JAVA接入示例,后面逐渐介绍其他接口其他语言

如何使用API接口查询物流信息?

电商行业的迅猛发展,推动着相关产业,尤其是物流行业需求的增加,据百度指数显示,针对物流查询的关键词搜索近年来始终居高不下并有不断上涨趋势,足见物流查询的需求之大.鉴于我在自身业务中的一些实际使用经验和总结,接下来就以用友APILink为例,讲一讲如何通过对接API接口,调用查询接口,获取物流信息.用友APILink官方平台上,"全国快递物流查询API"支持包括申通.顺丰.圆通.韵达.中通.汇通等快递公司在内的百家快递物流单号查询,是专门服务于快递查询的API接口.? 覆盖诸多快递公司:

2020最新的新浪短网址API接口分享-防封短网址如何生成分享

在短信,微信,微博,等等各大营销平台中我们都能看到http://t.cn/xxxx样式的新浪短链接,这种超短链接很好的满足了营销中要求链接简洁,美观,专业的特性,而且通过短网址包装后的链接对外发布也起到了防封的效果.那么新浪t.cn短链接如何生成呢?网址缩短API接口如何调用?今天我就给大家分享一下.   新浪t.cn短网址API接口: http://qingmeidwz.cn/shorten.php?url_long=http://www.baidu.com   微信url.cn短网址API接

调用API接口下载腾讯CDN访问日志

公司使用腾讯cdn为网站静态内容加速,由于业务需求,需要每天下载昨天的日志(因为腾讯方面给出回复,访问日志会有2个小时或以上时间的延迟,所以不建议下载当天日志,所以每天统计前一天的日志以做分析).因为cdn是由运维来管理,但是这个需求是业务的,如果每天都由运维进行下载,再通过邮件或其他工具发送,可能就显得麻烦.所幸腾讯CDN提供了API接口,因此采用shell脚本调用API进行下载的方式,定期下载日志,这样只要业务人员运行这个脚本就能自行下载日志,解放了运维的工作. #!/bin/bash ##

WebApi系列~通过HttpClient来调用Web Api接口

HttpClient是一个被封装好的类,主要用于Http的通讯,它在.net,java,oc中都有被实现,当然,我只会.net,所以,只讲.net中的HttpClient去调用Web Api的方法,基于api项目的特殊性,它需要有一个完全安全的环境,所以,你的api控制器看起来有点特别,只有5个方法,而且都是标准的http方法,我觉得这种设计很不错,很清晰,而且为了实现安全性,它不支持使用传统的表单数据,取而代之的是FromBody参数,它指拿HttpRequestMessage里参数,而不是所

JAVA调用聚合天气api接口示例

查询天气预报在APP中常用的一个常用功能,聚合数据免费天气api接口可以根据根据城市名/id查询天气.根据IP查询天气.据GPS坐标查询天气.查询城市天气三小时预报,并且支持全国不同城市天气预报查询. 代码描述:基于JAVA的免费天气api接口调用示例,根据文档中注明的需求参数,调用接口返回数据. 关联数据:免费天气api 接口地址:https://www.juhe.cn/docs/api/id/39 step1:选择本文所示例的接口"免费天气api" url:https://www.

C# 调用API接口处理公共类 自带JSON实体互转类

using System; using System.Collections.Generic; using System.IO; using System.Linq; using System.Net; using System.Net.Security; using System.Security.Cryptography.X509Certificates; using System.Text; using System.Threading.Tasks; using System.Web; n