how to copy pandas dataframe to vertica with airflow?

when I run the following function:

def write_df_to_vertica(conn: Cursor, df: pd.DataFrame):
    query = """COPY temp_data.test(coumn1, coumn2, coumn3) FROM STDIN DELIMITER ',' """
    conn.copy(query, df.to_csv(index=False, header=False))

it works, and the data is written to vertica db.

However when I call this function from PythonOperator in a AirFlow dag, the data isn't written to the db.

Why? and how to fix it?

1 answer

  • answered 2022-05-04 09:59 Rayan Suneer

    try this maybe?

                  name: str = "", 
                  schema: str = "", 
                  dtype: dict = {}, 
                  parse_n_lines: int = 10000, 
                  temp_path: str = "", 
                  insert: bool = False,)`

    this is the format for it i guess.

How many English words
do you know?
Test your English vocabulary size, and measure
how many words do you know
Online Test
Powered by Examplum