Remote SSH Command Execution Using CLI

See also questions close to this topic

• How to use the User role in Airflow 1.10 RBAC?

  I have been trying to look at how to use the User role. It says here, that it is for users with DAG ownership. So I created a couple of users with usernames ABC and XYZ and assigned them with User role.

  Here's my DAG:

  DEFAULT_ARGS = {
      'owner': 'ABC',
      ...,
      ...
  }
  
  dag = DAG(
      'test_dag',
      default_args=DEFAULT_ARGS,
      ...,
      ...
  )
  

  When I logged in as XYZ, I expected the DAG test_dag to be hidden. If not hidden then at least to be in inactive state, since test_dag belongs to ABC. But as XYZ, I'm able to operate test_dag.

  Am I missing anything out here?
• Airflow cron not running in timezone

  I have airflow setup with default_timezone = US/Eastern. If I set a DAG with schedule_interval="0 17 * * *" it runs at 12pm instead of the expected 5pm. I understand that airflow stores all dates as UTC. How can I get something to run in my timezone instead of writing the interval in UTC?

  I have also tried setting the tzinfo in the dag's start_date to pendulum.timezone('US/Eastern') with no luck.

  airflow=1.10.0 rhel7 python=3.6 server's tz = EST
• Celery executor in airflow without Backend

  When running airflow in a single node, can I configure the celeryexecutor without a Celery Backend (redis, rabbitmq, zookeeper) etc.

  Currently, since there is no backend infrastcucture available, running webserver and scheduler in single node (Localexecutor)
• Airflow 1.9.0 is queuing but tasks are not running

  Airflow stopped running tasks all of a sudden. Below are all running airflow scheduler airflow webserver airflow worker

  webui message

  All dependencies are met but the task instance is not running. In most cases this just means that the task will probably be scheduled soon unless: - The scheduler is down or under heavy load

  If this task instance does not start soon please contact your Airflow administrator for assistance.

  Scheduler seems to be in a loop, keeps repeating the below messages. WebUI shows tasks are in queued state. Tried restarting the scheduler, didn't help.

  [2018-11-17 22:03:45,809] {{jobs.py:1607}} DEBUG - Starting Loop...
  [2018-11-17 22:03:45,809] {{jobs.py:1627}} INFO - Heartbeating the process manager
  [2018-11-17 22:03:45,810] {{jobs.py:1662}} INFO - Heartbeating the executor
  [2018-11-17 22:03:45,810] {{base_executor.py:103}} DEBUG - 124 running task instances
  [2018-11-17 22:03:45,810] {{base_executor.py:104}} DEBUG - 0 in queue
  [2018-11-17 22:03:45,810] {{base_executor.py:105}} DEBUG - 76 open slots
  [2018-11-17 22:03:45,810] {{base_executor.py:132}} DEBUG - Calling the <class 'airflow.executors.celery_executor.CeleryExecutor'> sync method
  [2018-11-17 22:03:45,810] {{celery_executor.py:80}} DEBUG - Inquiring about 124 celery task(s)
  

  Airflow setup: apache-airflow[celery, redis, all]==1.9.0

  I also checked these posts but didn't help me:

  Airflow 1.9.0 is queuing but not launching tasks

  Airflow tasks get stuck at "queued" status and never gets running
• Xcom pull returns a NameError on `ti`

  I'm trying to push into xcom a value with key last_date in last_date_task and then pull it from xcom in second task ga_wh_task. when I test this dag by running python dag.py it returns an error at this line like:

      provide_context=True, since={{ti.xcom_pull(task_ids="last_date_task", key='last_date')}})
  NameError: name 'ti' is not defined
  

  What am I doing wrong here? I've tried using task_instance instead of ti and context['ti'] instead of kwargs['ti']

  Here's the dag file:

  default_args = {
      'owner': 'me',
      'start_date': dt.datetime(2017, 10, 30),
      'retries': 1,
      'retry_delay': dt.timedelta(minutes=10),
      'provide_context' : True,
  }
  
  
  
  def get_last_date(**kwargs):
      kwargs['ti'].xcom_push(key='last_date', value='2018-11-15')
      return True
  
  
  
  with DAG('ga_mysql_dag2',
           default_args=default_args,
           schedule_interval=None,
           catchup=False,
           ) as dag:
  
      last_date_task = PythonOperator(task_id='last_date_task', python_callable=get_last_date, provide_context=True)
  
      ga_wh_task = GoogleAnalyticsReportingToMySqlOperator(task_id='ga_wh_task', google_analytics_conn_id='google_analytics', key_file=key_file,\
                                          view_id=view_id, until=until, dimensions=dimensions, metrics=metrics, database=database,\
                                          table = table, mysql_conn_id = mysql_conn_id,
                                          provide_context=True, since={{ti.xcom_pull(task_ids="last_date_task", key='last_date')}})
      sleep = BashOperator(task_id='sleep', bash_command='sleep 10')
  
  # Dependencies
  last_date_task >> ga_warehouse_task >> sleep
  
• Is execution_date the date of the DAG run or the Task run?

  Is the value of execution_date the time date/time when the DAG ran--and is it the same value for all of its Tasks--or is execution_date (potentially) different per Task within a DAG?