我有一个装满了上千份文件的桶。我如何搜索水桶?


当前回答

这不是一个技术性的答案,但我已经构建了一个允许通配符搜索的应用程序:https://bucketsearch.net/

它将异步索引您的bucket,然后允许您搜索结果。

它是免费使用的(捐赠软件)。

其他回答

AWS发布了使用SQL: Amazon Athena https://aws.amazon.com/athena/查询S3桶的新服务

S3没有原生的“搜索此桶”,因为实际内容是未知的-此外,由于S3是基于键/值的,因此没有原生的方法可以一次访问多个节点,而更传统的数据存储提供了一个(SELECT * FROM…(在SQL模型中)。

您需要做的是执行ListBucket以获得bucket中对象的列表,然后遍历每个项,执行您实现的自定义操作—这就是您的搜索。

快进到2020年,使用aws-okta作为我们的2fa,下面的命令,尽管迭代这个特定bucket(+270,000)中的所有对象和文件夹非常缓慢,但运行良好。

aws-okta exec dev -- aws s3 ls my-cool-bucket --recursive | grep needle-in-haystax.txt

我做了如下的事情来在我的桶中寻找模式

def getListOfPrefixesFromS3(dataPath: String, prefix: String, delimiter: String, batchSize: Integer): List[String] = {
    var s3Client = new AmazonS3Client()
    var listObjectsRequest = new ListObjectsRequest().withBucketName(dataPath).withMaxKeys(batchSize).withPrefix(prefix).withDelimiter(delimiter)
    var objectListing: ObjectListing = null
    var res: List[String] = List()

    do {
      objectListing = s3Client.listObjects(listObjectsRequest)
      res = res ++ objectListing.getCommonPrefixes
      listObjectsRequest.setMarker(objectListing.getNextMarker)
    } while (objectListing.isTruncated)
    res
  }

对于较大的存储桶,这会消耗太多时间,因为所有对象摘要都由Aws返回,而不仅仅是匹配前缀和分隔符的对象。我正在寻找提高性能的方法,到目前为止,我只发现我应该命名键,并将它们正确地组织在桶中。

(至少)有两个不同的用例可以描述为“搜索桶”:

Search for something inside every object stored at the bucket; this assumes a common format for all the objects in that bucket (say, text files), etc etc. For something like this, you're forced to do what Cody Caughlan just answered. The AWS S3 docs has example code showing how to do this with the AWS SDK for Java: Listing Keys Using the AWS SDK for Java (there you'll also find PHP and C# examples). List item Search for something in the object keys contained in that bucket; S3 does have partial support for this, in the form of allowing prefix exact matches + collapsing matches after a delimiter. This is explained in more detail at the AWS S3 Developer Guide. This allows, for example, to implement "folders" through using as object keys something like folder/subfolder/file.txt If you follow this convention, most of the S3 GUIs (such as the AWS Console) will show you a folder view of your bucket.