DOCS
  • Saagie Documentation
    latest
      • Introduction
    • What’s New?
      • 2024
      • 2023
      • 2022
      • 2021
      • 2020
    • Getting Started Guide
      • Getting Familiar With the User Interface
      • Connecting to Saagie Using SSO For the First Time
      • Managing Your User Account
      • Starting With the Saagie Project Example
    • Data Team Guide
      • Projects Module
        • Projects
          • About Projects
          • Managing Projects
          • Project Settings
        • Jobs
          • About Jobs
          • Managing Jobs
          • Job Settings
        • Pipelines
          • About Pipelines
          • About Conditions in Pipelines
          • Managing Pipelines
          • Pipeline Settings
        • Apps
          • About Apps
          • Managing Apps
          • App Settings
        • Storage Spaces
          • About Storage Spaces
          • Managing Storage Spaces
        • Managing Environment Variables
        • Managing Docker Credentials
      • Monitoring Module
        • About the Monitoring Module
        • About Saagie Resources Monitoring
      • Catalog Module
        • About the Catalog Module
        • Repositories
          • About Repositories
          • Managing Repositories
          • Repository Settings
        • Technologies of the Official Saagie Repository
      • Security Module
        • About the Security Module
        • Managing User Accounts
        • Managing Groups
      • Add-On
        • Saagie Usage Monitoring
          • About Saagie Usage Monitoring
          • Saagie Usage Monitoring Default Dashboards
          • About the App Customization
        • Saagie Hugging Face Model Server
          • Using Saagie Hugging Face Model Server as an App
          • Using Saagie Hugging Face Model Server Via API
        • Saagie Code Search
          • About Saagie Code Search
          • Using Saagie Code Search
    • Developer Guide
      • Software Development Kit (SDK)
        • Creating and Managing Technologies
          • Creating the Metadata Files and the Zip Archive
          • Creating a Repository
          • References
        • About Docker Images Within Saagie
      • Application Programming Interface (API)
        • Saagie GraphQL API
        • Saagie Python API
      • Saagie CI/CD
        • Setting Up the CI/CD
        • Defining the CI/CD Stages
        • Using the CI/CD
        • References
    • Administrator Guide
      • About Saagie Architecture
        • How Does Saagie Work?
        • Interactions Between Saagie and External Components
        • Conf’o’rama
      • Installing and Setting Up Saagie
        • System Requirements
        • Creating and Configuring Your Kubernetes Cluster
          • Using Amazon Elastic Kubernetes Service (EKS)
          • Using Microsoft Azure Kubernetes Service (AKS)
          • Using Google Kubernetes Engine (GKE)
          • Using Another Service Platform
        • Downloading and Configuring Saagie
        • Running the Saagie Installer
        • Creating a Domain Name System (DNS) Entry
        • Enabling LDAP Authentication
        • Enabling Single Sign-On (SSO) Authentication
        • Your Commitments As a Self-Hosted Customer
      • Maintaining Your Platform
        • Upgrading Saagie
        • Updating Your SSL Certificate
        • Starting and Stopping Saagie
        • Adding a Node in Your Saagie Cluster
        • Changing the Data Lake URL in Saagie
        • Monitoring Technologies
        • Monitoring Logs
          • Audit Logs
          • Operational Logs
    • How-To Guides
      • Apache Sqoop
        • Import Data From a MySQL, Oracle, PostgreSQL, or SQL Server Database
        • Import Data From Other Relational Database Management System (RDBMS)
      • Apache Spark
        • Configure Spark Resources
        • Read and Write Files or Tables With PySpark
          • Read and Write Files From Amazon S3 Buckets With PySpark
          • Read and Write Files From HDFS With PySpark
          • Read and Write Tables From Hive With PySpark
        • Read and Write Files or Tables With Spark Scala
          • Read and Write Files From HDFS With Spark Scala
          • Read and Write Tables From Hive With Spark Scala
        • Package Your Spark Scala Code With the Assembly Plugin
        • Integrate Spark Streaming With Kafka
      • Airbyte
        • Airbyte Prerequisites
        • Use Airbyte As a Saagie App
        • Use Airbyte As an API
      • BigQuery
        • Create Machine Learning Pipelines With BigQuery in Saagie
          • Send Data to BigQuery
          • Preprocess Data (coming soon)
          • Create a Logistic Regression Model (coming soon)
          • Evaluate the Logistic Regression Model (coming soon)
      • HDFS
        • Manually Upload Large Files to HDFS
        • Read and Write Files From HDFS, WebHDFS, and HTTPFS With HDFS
      • Java/Scala
        • Read and Write Files or Tables With Java/Scala
          • Read and Write Files From HDFS With Java/Scala
          • Read and Write Tables From Hive With Java/Scala
          • Read and Write Tables From Impala With Java/Scala
          • Read and Write Files From MongoDB With Java/Scala
        • Use Java/Scala on Impala With High Availability
      • Kerberos
        • Adapt Your Job to Connect to Kerberos
      • Notebooks
        • Create RStudio User Accounts
        • Configure Git on RStudio
        • Use Generative AI in Jupyter Notebook
      • Python
        • Read and Write Files or Tables With Python
          • Read and Write Files From Amazon S3 Bucket With Python
          • Read and Write Files From Elasticsearch With Python
          • Read and Write Files From HDFS With Python
          • Read and Write Tables From Hive With Python
          • Read and Write Tables From Impala With Python
          • Read and Write Files From MongoDB With Python
          • Read and Write Tables From MySQL With Python
          • Read and Write Tables From PostgreSQL With Python
        • Use Python on HDFS or Impala With High Availability
      • R
        • Use R With a Proxy
        • Update R Packages
        • Use External R Packages
        • Create Hive Dynamic Tables
        • Read and Write Files or Tables With R
          • Read and Write Files From Amazon S3 Bucket With R
          • Read and Write Files From Elasticsearch With R
          • Read and Write Files From HDFS With R
          • Read and Write Tables From Hive With R
          • Read and Write Tables From Impala With R
          • Read and Write Files From MongoDB With R
        • Use R on HDFS With High Availability
        • Use R on Impala With High Availability
      • Talend
        • Read and Write Files or Tables With Talend
          • Read and Write Files From Amazon S3 Bucket With Talend
          • Read, Write, and List Files From HDFS With Talend
          • Read and Write Tables From Hive With Talend
          • Read and Write Tables From Impala With Talend
          • Read Files From MongoDB With Talend
        • Copy a File From HDFS to the Local Computer
        • Produce and Consume a Message in the Kafka Cluster
        • Use Talend on HDFS With High Availability
      • VS Code
        • Use VS Code Powered by Generative AI
      • Resume a Pipeline
      • Push Models to Hugging Face With Saagie
      • Launch GPU On Demand With OVH AI Training
    • FAQ
      • What are the basic Kubernetes commands I need to know to use Saagie?
      • How do I connect to a container?
      • How can I see the component logs?
      • How can I increase the storage size of a Persistent Volume Claim (PVC)?
      • How do I restore a component?
      • How can I contact Saagie support?
      • How do I request a feature?
      • Glossary
      • Site Map
  • How-To Guides
  • Apache Spark

Apache Spark

This section describes how to use Apache Spark with Saagie.

In this section:

  • Configure Spark Resources

  • Read and Write Files or Tables With PySpark

  • Read and Write Files or Tables With Spark Scala

  • Package Your Spark Scala Code With the Assembly Plugin

  • Integrate Spark Streaming With Kafka

Import Data From Other Relational Database Management System (RDBMS) Configure Spark Resources
Contact Support

This page was built using the Antora default UI.

The source code for this UI is licensed under the terms of the MPL-2.0 license.

×