GH-32123: [R] Expose azure blob filesystem#49553
Open
marberts wants to merge 88 commits intoapache:mainfrom
Open
GH-32123: [R] Expose azure blob filesystem#49553marberts wants to merge 88 commits intoapache:mainfrom
marberts wants to merge 88 commits intoapache:mainfrom
Conversation
We'll want to use the proper build flag before submitting the PR but for now this successfully links the C++ binding to the azurefs_is_functional_test function.
…rts/arrow into 32123-expose-azure-blob-filesystem
Note: I think what's happening is that the build of Arrow I install doesn't have the ARROW_AZURE flag enabled. When I "force" the environment variable I get an error like "fs___AzureFileSystem__Make not found".
Note: this was causing a seg fault when anything that inherits from FileSystem ries to access the base_path property. Uncommenting the return line and rebuilding the R package resolved this issue.
At this point, I was able to call write_feather(example_data, fs$path("test/test.feather")) successfully against Azurite.
Committing progress before I make any further changes.
…y with URI with Azure.
…ere skipped because of the URI issue.
Note: this was causing a seg fault when anything that inherits from FileSystem ries to access the base_path property. Uncommenting the return line and rebuilding the R package resolved this issue.
At this point, I was able to call write_feather(example_data, fs$path("test/test.feather")) successfully against Azurite.
Committing progress before I make any further changes.
…y with URI with Azure.
…ere skipped because of the URI issue.
Switch over to locally hosted azurite instead of local docker container azurite.
…ll the azurite process in a cleanup step.
rebuild docs with devtools::document(). rename argument in az_container to container_path instead of bucket
…ystem$create method
…rts/arrow into 32123-expose-azure-blob-filesystem
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
This PR adds support for Azure. The Arrow R package already has support for AWS and GCS, and the Arrow C++ library has had support for Azure for a couple years now. Support for Azure is already available in pyarrow.
This would close #32123.
What changes are included in this PR?
A new class
AzureFileSystemthat's analogous toS3FileSystem/GcsFileSystem, along with a helper functionaz_container()that's analogous tos3_bucket()/gcs_bucket().Updates to
src/filesystem.cppto interact with the machinery inarrow/filesystem/azurefs.h.Updates to the configuration and build scripts to support building with support for Azure.
Updates to the vignettes on cloud storage, installation, and developer setup.
Are these changes tested?
Yes. See
tests/testthat/test-azure.R.Are there any user-facing changes?
Yes. There is a new function
az_container(), serving the analogous role tos3_bucket()/gcs_bucket(), along with an R6 classAzureFileSystem, again serving the same role asS3FileSystemandGcsFileSystem. There is also a functionarrow_with_azure()to indicate if Arrow was built with support for Azure.