Enhancement Request: List keys (files) in S3 bucket using paginators

[adrianyorke]

I would like to connect to an existing S3 bucket and list the keys (files). This will then drive further test cases on a key-by-key basis as we fetch files in order and perform data quality checks such as row counts, hash value, average or total of numerical columns, etc..

There is a good write-up here which explains how this could be implemented using paginators. Paginators simplify paging complexities for buckets that contain large amounts of files.

[teaglebuilt]

this is a good place to start

[teaglebuilt]

if know one has claimed this issue. I will take it on

[adrianyorke]

Go for it @teaglebuilt. I will review/test and merge your pull request if that method works for you?

[adrianyorke]

Regarding PRs - would you prefer that we create a branch for each patch or are you happy just to work on master for now? Bigger projects (like robot framework core) normally prefer a separate branch for each patch but smaller projects can just work on master for simple fixes and enhancements.

[teaglebuilt]

i think we should always create pull requests.setting master as the tracking branch works instead of creating a dev branch for now. Maybe this will change when we have caught up. As far as documentation is concerned, i dont want the wait of pul request to hold back errors or incorrect documentation. I definately do not want to work off master from this point forward. If you are a collaborator, then you should be able to change incorrect information / documentation without a pull request. Like all of the markdown documentation, keyword docs, and so on

[teaglebuilt]

I have used robots libdoc for auto documentation, pre commit for linting, and I will set up travis ci to deploy to pypi on tag releases

[teaglebuilt]

after starting playing around with paginators, i think we need to think about all the s3 keywords that we want to create using this. This issue keyword should list all by page, or by prefix?

preferred keyword name?
other keyword offsets? For example:

List Keys params: bucket, prefix ?

Or

List Keys Bucket
List Keys by Prefix Bucket Prefix

[adrianyorke]

List Keys is what I had in mind. Should match the boto function. One thing to consider is that there may be many 1000s of keys in a single bucket so filtering option would be useful.

[teaglebuilt]

alright

[NeoMorfeo]

Any update on this? required help to develop?

[adrianyorke]

@NeoMorfeo: @teaglebuilt commented 15 Dec: "if know one has claimed this issue. I will take it on".