Software Engineer, Advanced

We are seeking a highly skilled Big Data Engineer to design, develop, and optimize scalable data pipelines and ETL processes. In this role, you will leverage technologies like Composer, Matillion and Spark, alongside strong expertise in SQL/OLAP databases (e.g., BigQuery), Object Storage, and messaging services. You will collaborate closely with cross-functional teams to deliver robust data solutions, utilizing IaC tools like Terraform and cloud platforms (preferably GCP). The ideal candidate brings strong programming skills (Python, Java, or Scala) and thrives in a collaborative, innovative environment focused on cutting-edge data architecture.

Key Responsibilities:

Big Data Processing:

• Design, develop, and maintain scalable data pipelines using technologies such as Apache Hadoop, Spark, and Kafka.
• Implement ETL (Extract, Transform, Load) processes to ingest, clean, and process large volumes of data from various sources.
• Optimize data processing workflows to ensure high performance and efficiency.
• Should have strong knowledge on SQL/OLAP databases like Big Query, or similar like Hive.
• Should have strong knowledge on Object Storage like Google Storage, or similar like S3, HDFS.
• Should have strong knowledge on Messaging services like Pub/Sub, or similar like SNS, Kafka maybe RabbitMQ.
• Hands on Experience in Iaas/PaaS service provisioning and automation tools like Terraform.
• Good to have knowledge on Cloud functions, Cloud build/Jenkins, Cloud Networking

Collaboration and Communication:

• Work closely with data scientists, engineers, and business analysts to understand data requirements and deliver solutions.
• Participate in code reviews, provide constructive feedback, and ensure coding standards are met.
• Communicate effectively with team members and stakeholders to ensure project goals and timelines are met.

Qualifications:

• Bachelor’s or Master’s degree in Computer Science, Information Technology, Data Science, or a related field.
• Proven experience in Big Data technologies such as Hadoop, Spark, Kafka, etc.
• Strong programming skills in languages such as Python, Java, or Scala.
• Familiarity with cloud platforms (e.g. Google Cloud) and their data processing services.
• Excellent problem-solving skills and attention to detail.
• Strong communication and teamwork abilities.

Preferred Qualifications:

• Experience with containerization and orchestration tools like Docker and Kubernetes.
• Familiarity with CI/CD pipelines and version control systems like Git.
• GCP Knowledge is more preferable.

• • Company Culture: Our company values innovation, collaboration, and continuous learning. We offer a dynamic work environment where you can contribute to cutting-edge projects and grow your skills in cloud-based ETL testing.