12/5/2023 0 Comments Airflow kubernetes operator![]() ![]() MainApplicationFile: "local:///opt/spark/examples/jars/spark-examples_2.12-3.4.0. # See the License for the specific language governing permissions andĪpiVersion: "/v1beta2" # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # distributed under the License is distributed on an "AS IS" BASIS, # Unless required by applicable law or agreed to in writing, software # You may obtain a copy of the License at # you may not use this file except in compliance with the License. Sample YAML file : # Copyright 2017 Google LLC # Licensed under the Apache License, Version 2.0 YAML file will create driver and executor pod to run spark jobĪirflow Task: spark_operator = SparkKubernetesOperator(Īpplication_file='spark_app_config.yaml', In airflow we can use "SparkKubernetesOperator" and provide spark job details in ".yaml" file.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |