2
VOTE
Status
Future consideration
Workspace
* IBM Cloud Ideas
Categories
Databases
Created by
Guest
Created on
Sep 12, 2024
RELATED IDEAS
Support for Parquet S3 Foreign Data Wrapper for PostgreSQL for psql PaaS
Please add support for parquet S3 FDW for s3 server ad external data wrapper.
It's been officially released on Dic 23
https://www.postgresql.org/about/news/parquet-s3-fdw-110-released-2768/
but right now it's not an extension supported by IBM Cloud.
The installation guide is at this repo
https://github.com/pgspider/parquet_s3_fdw
For example, we are now in the need to dump a table to a csv file on an external storage (best if s3, but not mandatory).
We are unable to use any native extension, since the supported fdw are just file (how can I use it when I don't have access to the local disk of the server?) and the postgres (but I don't need an external psql db).
So we will go with some psql client or driver and ibmcloud cli/sdk to achieve the same result as we would have with this extension.
| Idea priority | Medium |
| Needed By | Yesterday (Let's go already!) |
By clicking the "Post Comment" or "Submit Idea" button, you are agreeing to the IBM Ideas Portal Terms of Use.
Do not place IBM confidential, company confidential, or personal information into any field.
Thank you!
Hello Venkat,
I just send a comment, but I don't it published.
You are absolutely right, we don't need exactly the parquet extension, but we need at least a fwd for S3, just like https://fdw.dev/catalog/s3/
But the wrapper extension is not present on IBM Cloud psql (https://cloud.ibm.com/docs/databases-for-postgresql?topic=databases-for-postgresql-extensions#available-extensions)
We are running a SaaS service on k8s and need to dump some tables for a customer to reach, and would like to enable him through S3
We have Liberty and OpenLiberty microservices, and we would prefer to export these tables with Java rather than with spin up a pod with the pg_dump installed on it.
Thanks
Eugenio
Hello Venkat,
I just send a comment, but I don't it published.
You are absolutely right, we don't need exactly the parquet extension, but we need at least a fwd for S3, just like https://fdw.dev/catalog/s3/
But the wrapper extension is not present on IBM Cloud psql (https://cloud.ibm.com/docs/databases-for-postgresql?topic=databases-for-postgresql-extensions#available-extensions)
We are running a SaaS service on k8s and need to dump some tables for a customer to reach, and would like to enable him through S3
We have Liberty and OpenLiberty microservices, and we would prefer to export these tables with Java rather than with spin up a pod with the pg_dump installed on it.
Thanks
Eugenio
Hello Venkat,
yes, sure.
We are running a SaaS service on kubernetes and we need to periodically export some tables for a customer outside IBM Cloud.
We would like to enable him to an S3 Bucket, where he can find the exports without the need to use a jump host (the db has only private endpoints).
We would avoid, if possible, the pg_dump, since we are using Liberty and Openliberty webserver, so we would like to dump data via Java, rather than spinning up a pod with pg_dump installed on it.
This said, you are absolutely right, it doesn't need to be dedicated to parquet file, even an s3_wrapper (like https://fdw.dev/catalog/s3/) is fine.
But it needs the wrapper extension, which is currently not available (https://cloud.ibm.com/docs/databases-for-postgresql?topic=databases-for-postgresql-extensions)
So it would be feasible to add a fdw for s3 as an extension?
Thanks
Eugenio
Hi Eugenio,
Thank you for your patience.
I am looking for some more details. Could you please explain this use case?
Could you explain why you are requesting a Parquet export extension when you stated that the requirement is for CSV export? Specifically, why is pg_dump for CSV inadequate for your needs, particularly if your goal is to externalize tables to a storage or local file system?
Thank you, and I look forward to hearing from you.
Which kind of more information do you need?
Thanks