Summary
HCLTech’s posting is titled Senior Technical Lead and describes a PySpark Developer role in Dubai. The role involves designing and maintaining scalable ETL/ELT pipelines, processing large datasets, tuning Spark jobs, and contributing to data models and storage formats. It also involves connecting Spark applications to cloud storage and databases, deploying workflows, and maintaining code quality through unit testing. The posting asks for advanced Python and SQL skills and experience with big data frameworks, cloud platforms, storage formats, and orchestration tools.
The posting
As published on hcltech.com. Pulled from the employer's site and shown as they wrote it.
Job Summary
A PySpark Developer is responsible for designing, developing, and maintaining large-scale, distributed data processing systems and pipelines. They leverage Apache Spark’s Python API (PySpark) to ingest, transform, and analyze massive volumes of structured and unstructured data.
Core Responsibilities
- Data Pipeline Development: Design and build scalable, efficient ETL/ELT pipelines to move and transform data from various sources into data lakes or warehouses.
- Big Data Processing: Write optimized PySpark scripts (using DataFrames, Datasets, and Spark SQL) to process terabytes/petabytes of data.
- Performance Tuning: Troubleshoot, debug, and optimize Spark jobs by managing partitions, caching, broadcast variables, and minimizing data shuffling.
- Data Modeling: Assist in designing data architectures, schemas, and storage formats (e.g., Parquet, Delta Lake, ORC).
- Integration: Connect Spark applications with cloud storage (AWS S3, Azure ADLS, Google Cloud Storage) and databases (NoSQL, relational).
- CI/CD & Automation: Deploy data workflows using orchestration tools and maintain code quality through unit testing.
Technical Skills Required
Category
Skills & Technologies
Core Languages
Python (Advanced), SQL (Advanced)
Big Data Frameworks
Apache Spark / PySpark, Hadoop, Hive
Cloud Platforms
AWS (EMR, Glue), Azure (Databricks, Synapse), or GCP (Dataproc)
Storage & Formats
Delta Lake, Parquet, Avro, HDFS, Snowflake, Amazon S3
Orchestration
Apache Airflow, Prefect, or Luigi
Key Responsibilities
as above
Skill Requirements
as above