قائد تقني أول

Job Summary

A PySpark Developer is responsible for designing, developing, and maintaining large-scale, distributed data processing systems and pipelines. They leverage Apache Spark’s Python API (PySpark) to ingest, transform, and analyze massive volumes of structured and unstructured data.

Core Responsibilities

• Data Pipeline Development: Design and build scalable, efficient ETL/ELT pipelines to move and transform data from various sources into data lakes or warehouses.
• Big Data Processing: Write optimized PySpark scripts (using DataFrames, Datasets, and Spark SQL) to process terabytes/petabytes of data.
• Performance Tuning: Troubleshoot, debug, and optimize Spark jobs by managing partitions, caching, broadcast variables, and minimizing data shuffling.
• Data Modeling: Assist in designing data architectures, schemas, and storage formats (e.g., Parquet, Delta Lake, ORC).
• Integration: Connect Spark applications with cloud storage (AWS S3, Azure ADLS, Google Cloud Storage) and databases (NoSQL, relational).
• CI/CD & Automation: Deploy data workflows using orchestration tools and maintain code quality through unit testing.

Technical Skills Required

Category

Skills & Technologies

Core Languages

Python (Advanced), SQL (Advanced)

Big Data Frameworks

Apache Spark / PySpark, Hadoop, Hive

Cloud Platforms

AWS (EMR, Glue), Azure (Databricks, Synapse), or GCP (Dataproc)

Storage & Formats

Delta Lake, Parquet, Avro, HDFS, Snowflake, Amazon S3

Orchestration

Apache Airflow, Prefect, or Luigi

Key Responsibilities

as above

Skill Requirements

as above