GH-49566: [Python] Skip header files when installing compiled Cython files by AlenkaF · Pull Request #49571 · apache/arrow

AlenkaF · 2026-03-20T10:20:16Z

Rationale for this change

Local import is broken when doing editable install with scikit-build-core, also nightly verification jobs are failing for the same reason.

What changes are included in this PR?

lib.h and lib_api.h are already installed separately so we skip them when installing Cython extensions into the output destination (side-packages/pyarrow).

Are these changes tested?

Yes, locally and via the extended verification builds.

Are there any user-facing changes?

No.

GitHub Issue: [Python] Cython API header files seem to break editable installs #49566

github-actions · 2026-03-20T10:21:29Z

⚠️ GitHub issue #49566 has been automatically assigned in GitHub to PR creator.

AlenkaF · 2026-03-20T10:22:52Z

@github-actions crossbow submit verify-rc-source-python-*

github-actions · 2026-03-20T10:25:23Z

Revision: 719eaf2

Submitted crossbow builds: ursacomputing/crossbow @ actions-3593c891fd

Task	Status
verify-rc-source-python-linux-almalinux-10-amd64
verify-rc-source-python-linux-conda-latest-amd64
verify-rc-source-python-linux-ubuntu-22.04-amd64
verify-rc-source-python-linux-ubuntu-24.04-amd64
verify-rc-source-python-macos-amd64
verify-rc-source-python-macos-arm64
verify-rc-source-python-macos-conda-amd64

raulcd · 2026-03-20T10:47:12Z

macOS failure are unrelated, I am trying to find a fix, the issue tracking them is this one:

[CI][Python][C++] macos 14 fails with missing std::log2p1 #49569

AlenkaF · 2026-03-20T10:49:52Z

We have a different issue now:

_____________________________ test_pyarrow_include _____________________________

    def test_pyarrow_include():
        # We need to make sure that pyarrow/include is always
        # created. Either with PyArrow C++ header files or with
        # Arrow C++ and PyArrow C++ header files together
    
        source = os.path.dirname(os.path.abspath(__file__))
        pyarrow_dir = pjoin(source, '..')
        pyarrow_include = pjoin(pyarrow_dir, 'include')
        pyarrow_cpp_include = pjoin(pyarrow_include, 'arrow', 'python')
    
>       assert os.path.exists(pyarrow_include)
E       AssertionError: assert False
E        +  where False = <function exists at 0x7ff433ef9e80>('/arrow/python/pyarrow/tests/../include')
E        +    where <function exists at 0x7ff433ef9e80> = <module 'posixpath' (frozen)>.exists
E        +      where <module 'posixpath' (frozen)> = os.path

Investigating ...

AlenkaF · 2026-03-20T11:11:20Z

@raulcd I have updated the failing test as it was still checking the old location of the include folder which now moved to the site-packages. What do you think? The commit: 9ba4a76

AlenkaF · 2026-03-20T11:11:38Z

@github-actions crossbow submit verify-rc-source-python-*

github-actions · 2026-03-20T11:14:01Z

Revision: 9ba4a76

Submitted crossbow builds: ursacomputing/crossbow @ actions-7e817c381a

Task	Status
verify-rc-source-python-linux-almalinux-10-amd64
verify-rc-source-python-linux-conda-latest-amd64
verify-rc-source-python-linux-ubuntu-22.04-amd64
verify-rc-source-python-linux-ubuntu-24.04-amd64
verify-rc-source-python-macos-amd64
verify-rc-source-python-macos-arm64
verify-rc-source-python-macos-conda-amd64

AlenkaF · 2026-03-20T11:45:54Z

Another issue I need to investigate:

arrow/dev/release/verify-release-candidate.sh: line 537: 94675 Bus error: 10              python -c "import pyarrow.flight"

AlenkaF · 2026-03-20T11:53:06Z

@github-actions crossbow submit verify-rc-source-python-macos-arm64

AlenkaF · 2026-03-20T11:53:25Z

Testing if it is a flaky macos issue ...

github-actions · 2026-03-20T11:55:20Z

Revision: 9ba4a76

Submitted crossbow builds: ursacomputing/crossbow @ actions-49b32e9b3a

Task	Status
verify-rc-source-python-macos-arm64

AlenkaF · 2026-03-20T15:33:59Z

@github-actions crossbow submit verify-rc-source-python-macos-amd64

github-actions · 2026-03-20T15:36:26Z

Revision: 9ba4a76

Submitted crossbow builds: ursacomputing/crossbow @ actions-109f022a4d

Task	Status
verify-rc-source-python-macos-amd64

AlenkaF · 2026-03-23T07:29:42Z

verify-rc-source-python-macos-arm64 failure seems to be connected to #49569, but verify-rc-source-python-macos-amd64 is failing due to:

arrow/dev/release/verify-release-candidate.sh: line 537: 70649 Bus error: 10              python -c "import pyarrow.flight"

cc @raulcd for ideas? (can't test locally, my chip is arm)

…new one for the build

raulcd · 2026-03-24T10:18:38Z

@github-actions crossbow submit verify-rc-source-python-macos-amd64 verify-rc-source-python-macos-arm64

raulcd · 2026-03-24T10:20:37Z

@AlenkaF I've rebased main (the macOS arm64 should be fixed there, just validating now) and have pushed a new commit because I think the verification wasn't correct. Is not using the built virtualenv/conda env. I am unsure what the macOS Intel failure is but the reason doesn't seem to be the same one as the one for the other failures so we might want to tackle that one on a separate issue.

github-actions · 2026-03-24T10:20:55Z

Revision: 3f0f110

Submitted crossbow builds: ursacomputing/crossbow @ actions-b16e795f8d

Task	Status
verify-rc-source-python-macos-amd64
verify-rc-source-python-macos-arm64

AlenkaF · 2026-03-24T10:55:07Z

Python 3.14 verification on macos arm is now failing due to:

pip._vendor.pyproject_hooks._impl.BackendUnavailable: Cannot import '_build_backend'

update: both builds have the same, above, issue now 👍

raulcd · 2026-03-24T11:42:12Z

@github-actions crossbow submit verify-rc-source-python-macos-amd64 verify-rc-source-python-macos-arm64

github-actions · 2026-03-24T11:44:28Z

Revision: f7d1f9b

Submitted crossbow builds: ursacomputing/crossbow @ actions-eb246d1d18

Task	Status
verify-rc-source-python-macos-amd64
verify-rc-source-python-macos-arm64

raulcd · 2026-03-24T12:28:32Z

@github-actions crossbow submit verify-rc-source-python-macos-amd64

github-actions · 2026-03-24T12:30:46Z

Revision: b865cd5

Submitted crossbow builds: ursacomputing/crossbow @ actions-9b5afd26a7

Task	Status
verify-rc-source-python-macos-amd64

…e environment

raulcd · 2026-03-24T16:39:38Z

@github-actions crossbow submit verify-rc-source-python-macos-amd64

github-actions · 2026-03-24T16:42:10Z

Revision: 3ec70f6

Submitted crossbow builds: ursacomputing/crossbow @ actions-aad2e8bd9a

Task	Status
verify-rc-source-python-macos-amd64

raulcd · 2026-03-24T17:11:37Z

@github-actions crossbow submit verify-rc-source-python-macos-amd64

github-actions · 2026-03-24T17:13:53Z

Revision: 8918219

Submitted crossbow builds: ursacomputing/crossbow @ actions-6df01d3341

Task	Status
verify-rc-source-python-macos-amd64

…ow_flight.so

raulcd · 2026-03-24T17:48:07Z

@github-actions crossbow submit verify-rc-source-python-macos-amd64 verify-rc-source-python-macos-arm64

github-actions · 2026-03-24T17:50:24Z

Revision: 9ef21fd

Submitted crossbow builds: ursacomputing/crossbow @ actions-c945e24f25

Task	Status
verify-rc-source-python-macos-amd64
verify-rc-source-python-macos-arm64

raulcd · 2026-03-24T18:51:13Z

@github-actions crossbow submit verify-rc-source-python-macos-amd64

github-actions · 2026-03-24T18:53:26Z

Revision: 5f8c18c

Submitted crossbow builds: ursacomputing/crossbow @ actions-e5fbc461ef

Task	Status
verify-rc-source-python-macos-amd64

raulcd · 2026-03-24T19:44:35Z

@github-actions crossbow submit verify-rc-source-python-macos-amd64

github-actions · 2026-03-24T19:46:02Z

502 Server Error
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/23509011623

raulcd · 2026-03-24T20:35:55Z

@github-actions crossbow submit verify-rc-source-python-macos-amd64

github-actions · 2026-03-24T20:38:14Z

Revision: a5ff56d

Submitted crossbow builds: ursacomputing/crossbow @ actions-0c439b7c1c

Task	Status
verify-rc-source-python-macos-amd64

kou · 2026-03-24T21:51:30Z

dev/release/verify-release-candidate.sh

  fi
  if [ "${ARROW_FLIGHT}" == "ON" ]; then
-    python -c "import pyarrow.flight"
+   python -c "


Suggested change

python -c "

python -c "

AlenkaF requested review from raulcd and rok as code owners March 20, 2026 10:20

github-actions bot added Component: Python awaiting review Awaiting review labels Mar 20, 2026

AlenkaF and others added 3 commits March 24, 2026 11:17

Skip header files when installing compiled Cython files

41026d4

Fix old build include location test

efc1ae9

Use the virtualenv / conda env created instead of trying to create a …

3f0f110

…new one for the build

raulcd force-pushed the gh-49566-cython-headers branch from 9ba4a76 to 3f0f110 Compare March 24, 2026 10:18

raulcd requested review from assignUser, jonkeane and kou as code owners March 24, 2026 10:18

The virtualenv must contain the requirements for building

f7d1f9b

Trying to diagnose crash importing flight on macOS intel

b865cd5

raulcd added 2 commits March 24, 2026 14:16

Verification jobs shouldn't need to require editable builds

612c41b

Avoid testing on the python folder to use the installed pyarrow on th…

3ec70f6

…e environment

Add a bunch more debugging for macOS intel

8918219

Append _arrow_lib_dir to RPATH so _flight.so (python) can find libarr…

9ef21fd

…ow_flight.so

More debugging

5f8c18c

Fix protobug 33 and 34 on the same process

a5ff56d

kou reviewed Mar 24, 2026

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Mar 24, 2026

Conversation

AlenkaF commented Mar 20, 2026 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions bot commented Mar 20, 2026

Uh oh!

AlenkaF commented Mar 20, 2026

Uh oh!

github-actions bot commented Mar 20, 2026

Uh oh!

raulcd commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AlenkaF commented Mar 20, 2026

Uh oh!

AlenkaF commented Mar 20, 2026

Uh oh!

AlenkaF commented Mar 20, 2026

Uh oh!

github-actions bot commented Mar 20, 2026

Uh oh!

AlenkaF commented Mar 20, 2026

Uh oh!

AlenkaF commented Mar 20, 2026

Uh oh!

AlenkaF commented Mar 20, 2026

Uh oh!

github-actions bot commented Mar 20, 2026

Uh oh!

AlenkaF commented Mar 20, 2026

Uh oh!

github-actions bot commented Mar 20, 2026

Uh oh!

AlenkaF commented Mar 23, 2026

Uh oh!

raulcd commented Mar 24, 2026

Uh oh!

raulcd commented Mar 24, 2026

Uh oh!

github-actions bot commented Mar 24, 2026

Uh oh!

AlenkaF commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

raulcd commented Mar 24, 2026

Uh oh!

github-actions bot commented Mar 24, 2026

Uh oh!

raulcd commented Mar 24, 2026

Uh oh!

github-actions bot commented Mar 24, 2026

Uh oh!

raulcd commented Mar 24, 2026

Uh oh!

github-actions bot commented Mar 24, 2026

Uh oh!

raulcd commented Mar 24, 2026

Uh oh!

github-actions bot commented Mar 24, 2026

Uh oh!

raulcd commented Mar 24, 2026

Uh oh!

github-actions bot commented Mar 24, 2026

Uh oh!

raulcd commented Mar 24, 2026

Uh oh!

github-actions bot commented Mar 24, 2026

Uh oh!

raulcd commented Mar 24, 2026

Uh oh!

github-actions bot commented Mar 24, 2026

Uh oh!

raulcd commented Mar 24, 2026

Uh oh!

github-actions bot commented Mar 24, 2026

Uh oh!

AlenkaF commented Mar 20, 2026 •

edited by github-actions bot

Loading

raulcd commented Mar 20, 2026 •

edited

Loading

AlenkaF commented Mar 24, 2026 •

edited

Loading