Beautiful post series, I've reading a lot of documentation lately and your articles are by far the most serious in this.
I'd like to know a few things also:
Do you make use of the spark crawlers moving on your raw zone? My idea was to make use of dynamic frames against s3 and resolve schemas conflicts prior converting the data to dataframes.
Would you include support for the latest aws glue docker image? I'm currently using it and allowed me to launch the aws glue tests.
Hi Adrian,
We did not make use of crawlers for moving data from raw zone. Let me know how your idea went.
I'm aware of the latest glue container, I did not have the opportunity to try it out yet but I can include it as soon as I do because it's quite important part of the dev flow.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Beautiful post series, I've reading a lot of documentation lately and your articles are by far the most serious in this.
I'd like to know a few things also:
Thans for everything
Hi Adrian,
We did not make use of crawlers for moving data from raw zone. Let me know how your idea went.
I'm aware of the latest glue container, I did not have the opportunity to try it out yet but I can include it as soon as I do because it's quite important part of the dev flow.