Data Engineer – Executive Opening at Annalect in Bengaluru

• Partner with application architects and developers to build the data infrastructure used by our products.
• Assist Project Managers, Business Analysts, and Tech Lead in project planning (provide estimates, risk analysis, requirements analysis, technical options)
• Optimize operations against large databases.
• Help define the vision for a product, identify dead ends and routes with untapped potential.
• Break things, rebuild them, make them better, and share with the wider team your learnings.
• Involvement in full life cycle of projects (including requirement analysis and system design, development and support if required)
• Support and work collaboratively with teams across areas of design, development, quality assurance and operations
• Commit your knowledge and experience into team success
• A small amount of travel (when needed) to work with other offices
• Attend meetings to discuss and demonstrate solutions at both technical and business level Steward data and compute environments to facilitate usage of data assets

Responsibilities

Designing, building, testing and deploying scalable, reusable and maintainable applications that handle large amounts of data.

•     Write at-scale ETL processes in Python, Spark, and other technologies.

•     Perform code reviews and provide leadership and guidance to junior developers.

•     Ability to learn and teach new technologies.

•     Curiosity in learning the business requirements that are driving the engineering requirements.

•     Interest in new technologies and eager to bring those technologies and out of the box ideas to the team.

•     4+ years of development experience on web applications using Python, Ruby, Java, or C#.

•     4+ years of SQL experience.

•     Intellectual curiosity and drive; self-starters will thrive in this position.

•     Passion for Technology: Excitement for new technology, bleeding edge applications, and a positive attitude towards solving real world challenges.

Qualifications

BS, MS or PhD in Computer Science, Engineering, or equivalent real-world experience (You’ve learned something / somehow to be able to claim you are an engineer)

•     Significant experience with Python, C++, or other popular language

•     Experience with big data and/or infrastructure. Bonus for having experience in setting up Petabytes of data so they can be easily accessed. Understanding of data organization, i.e., partitioning, clustering, file sizes, file formats. Data cataloging with Hive/Hive metastore or Glue or something similar.

•     Experience working with classical relational databases.

•     Experience with Hadoop, Hive, Spark, or other data processing tools (Lots of time will be spent building and optimizing transformations)

•     Experience building scalable data pipelines (Airflow experience a plus)

•     Significant experience working with AWS and/or GCP

•     Proven ability to independently execute projects from concept to implementation to launch and to maintain a live product

•     Interest or experience in ML technologies (TensorFlow, PyTorch or SageMaker, BigQueryML).