Skip to main content
EPAM Systems logo

Lead Data Engineer (PySpark, statistics, MLflow)

EPAM Systems
1 day ago
Remote
Romania
Data Science and Analytics

We are looking for a Lead Data Engineer to join our team and take ownership of complex data engineering challenges, from migrating legacy codebases to building robust, production-grade pipelines. In this role, you will work independently on large-scale PySpark systems, apply statistical methods to real-world sensor data, and leverage MLflow to manage the full lifecycle of machine learning models.

Responsibilities

  • Migrate existing pandas, Koalas or pandas-on-Spark code to native PySpark implementations
  • Decompose and refactor large legacy notebooks without relying on existing documentation
  • Apply statistical techniques to detect outliers and events in sensor time series data
  • Package and manage machine learning models using MLflow, including experiment tracking, Model Registry and versioning
  • Implement model explainability techniques such as SHAP for existing models
  • Develop unit and regression tests for PySpark pipelines using pytest and tools like assertDataFrameEqual or chispa
  • Work independently on data engineering projects with minimal oversight
  • Integrate AI coding assistants into daily workflows and critically review their output
  • Utilize Window functions to optimize data transformation logic

Requirements

  • 5+ years of experience in data engineering with deep knowledge of PySpark and pandas
  • Proven experience migrating pandas, Koalas or pandas-on-Spark code to native PySpark, including Window functions
  • Capability to read and decompose large legacy notebooks without documentation, and to work independently on them
  • Expertise in applied statistics, including outlier detection and event detection in sensor time series
  • Proficiency in MLflow for experiment tracking, Model Registry and versioning, with skills in packaging existing models including explainability techniques like SHAP
  • Skills in writing unit and regression tests for PySpark using pytest and assertDataFrameEqual or chispa
  • Familiarity with using AI coding assistants in daily work and reviewing their output critically
  • English proficiency at B2 level or higher

Nice to have

  • Knowledge of incremental processing on Delta, including MERGE and Change Data Feed, and Databricks job triggers
  • Background in working with sensor or IoT data
  • Databricks Machine Learning Professional certification
  • German proficiency
Job alerts

Not the right fit? Get jobs like this by email

New developer jobs, straight to your inbox. Instant alerts or a weekly digest.

Free. No spam, unsubscribe anytime.