Problems installing pyarrow in a virtual environment /u/Intentionalrobot Python Education

Context: I’m a data analyst and I usually work in only one environment that’s mainly Jupyter Notebooks. I don’t know anything about software best practices or development, github, and I have a tenuous grasp on coding in general.

My Goal: I recently built a simple AI Agent in python that connects my companies’ BigQuery database to an LLM and then outputs that AI response back into BigQuery.

I need to find a way to deploy this to google cloud so that my co-workers can interact with it. I decided I am going to use Streamlit, which is supposedly the easiest way to stand up a front end for a little Python app.

The Problem: I got a simple “hello world” streamlit page up, but when I try to recreate the environment to build my AI Agent in the new environment, the installation of key packages doesn’t work. Pyarrow is the main one I’m having trouble with right now.

I read online that I should create a virtual environment for deploying my app to the cloud. I’m not sure if this is strictly necessary, but that’s what I’ve been trying to do because I’m just following the steps. Plus, I couldn’t run streamlit from my jupyter notebooks.

What i’ve done: I created the virtual environment using python3 -m venv .venv, which works fine, but when I try to install the packages I need (like pyarrow, langchain, pandas, etc.), I keep running into errors. I expected that I would just create the environment, activate it, and then run pip install pyarrow, pip install langchain, and pip install pandas. However, instead of it installing smoothly, I started getting errors with pyarrow and ended up having to install things like cmake, apache-arrow, and more. But, it’s frustrating because none of these installations of cmake or apache-arrow are solving the problem with pyarrow.

Snippet of the Errors:

Collecting pyarrow

Using cached pyarrow-18.1.0.tar.gz (1.1 MB)

Installing build dependencies … done

Getting requirements to build wheel … done

Preparing metadata (pyproject.toml) … done

Building wheels for collected packages: pyarrow

Building wheel for pyarrow (pyproject.toml) … error

error: subprocess-exited-with-error

× Building wheel for pyarrow (pyproject.toml) did not run successfully.

│ exit code: 1

╰─> [832 lines of output]

— Configuring incomplete, errors occurred!

error: command ‘/usr/local/bin/cmake’ failed with exit code 1

[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.

ERROR: Failed building wheel for pyarrow

Failed to build pyarrow

ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (pyarrow)

________________________________________________

I’ve been trying to troubleshoot online, but nothing is really working.

Any help would be greatly appreciated. If you could point me toward the key concepts I need to understand in order to diagnose the issue, that would be really helpful. If you have any specific advice, I would love that.

Context: I’m a data analyst and I usually work in only one environment that’s mainly Jupyter Notebooks. I don’t know anything about software best practices or development, github, and I have a tenuous grasp on coding in general.

My Goal: I recently built a simple AI Agent in python that connects my companies’ BigQuery database to an LLM and then outputs that AI response back into BigQuery.

I need to find a way to deploy this to google cloud so that my co-workers can interact with it. I decided I am going to use Streamlit, which is supposedly the easiest way to stand up a front end for a little Python app.

The Problem: I got a simple “hello world” streamlit page up, but when I try to recreate the environment to build my AI Agent in the new environment, the installation of key packages doesn’t work. Pyarrow is the main one I’m having trouble with right now.

I read online that I should create a virtual environment for deploying my app to the cloud. I’m not sure if this is strictly necessary, but that’s what I’ve been trying to do because I’m just following the steps. Plus, I couldn’t run streamlit from my jupyter notebooks.

What i’ve done: I created the virtual environment using python3 -m venv .venv, which works fine, but when I try to install the packages I need (like pyarrow, langchain, pandas, etc.), I keep running into errors. I expected that I would just create the environment, activate it, and then run pip install pyarrow, pip install langchain, and pip install pandas. However, instead of it installing smoothly, I started getting errors with pyarrow and ended up having to install things like cmake, apache-arrow, and more. But, it’s frustrating because none of these installations of cmake or apache-arrow are solving the problem with pyarrow.

Snippet of the Errors:

Collecting pyarrow

Using cached pyarrow-18.1.0.tar.gz (1.1 MB)

Installing build dependencies … done

Getting requirements to build wheel … done

Preparing metadata (pyproject.toml) … done

Building wheels for collected packages: pyarrow

Building wheel for pyarrow (pyproject.toml) … error

error: subprocess-exited-with-error

× Building wheel for pyarrow (pyproject.toml) did not run successfully.

│ exit code: 1

╰─> [832 lines of output]

— Configuring incomplete, errors occurred!

error: command ‘/usr/local/bin/cmake’ failed with exit code 1

[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.

ERROR: Failed building wheel for pyarrow

Failed to build pyarrow

ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (pyarrow)

________________________________________________

I’ve been trying to troubleshoot online, but nothing is really working.

Any help would be greatly appreciated. If you could point me toward the key concepts I need to understand in order to diagnose the issue, that would be really helpful. If you have any specific advice, I would love that.