Software Engineer (Data Platform)
Mercari, Inc. (India), India
Job Description
Introduction
Circulate all forms of value to unleash the potential in all people
"What can I do to help society thrive with the finite resources we have?" The Mercari marketplace app was born in 2013 out of this thought by our founder Shintaro Yamada as he traveled the world. We believe that by circulating all forms of value, not just physical things and money, we can create opportunities for anyone to realize their dreams and contribute to society and the people around them. Mercari aims to use technology to connect people all over the world and create a world where anyone can unleash their potential.For more information about Mercari Group’s mission, see Mercari's Culture Doc.
Equal Opportunity Hiring
Here at Mercari, we work to realize a world in which no one’s potential is limited by their background and everyone has the opportunity to freely create value. We also firmly believe that a mindset of Diversity amp; Inclusion is essential for us to achieve our mission.
This, of course, extends to our hiring practices as well. Mercari is committed to eliminating discrimination based on age, gender, sexual orientation, race, religion, physical disability, and other such factors so that anyone who shares our mission and values can join us, regardless of their background. For more details, please read our Damp;I Statement.
Position Overview- Mercari Group utilizes data for business decisions and other processes in a wide variety of areas, including marketing, machine learning, and Ramp;D. As part of Merpay’s Data Platform Team, you will design, develop and operate the data infrastructure/data pipelines which support data utilization for the entire Mercari Group, including not only Merpay, but the Mercari marketplace app. You will also be responsible for work on the Mercari Group’s data utilization as a whole.
- The Data Platform Team is an engineering organization which builds the systems necessary to a wide range of domains that utilize data. This work includes development of an ecosystem enabling data collection, Data Lake, DWH, use of collected data and other processes required by data utilization. The following describes the responsibilities of a software engineer on the Data Platform Team, as well as the environment and organization in which they work:
Requirements
- Job Description
- Collecting data from microservices
- Developing batch-based data pipelines
- Developing streaming-based data pipelines
- Developing SDK for logging within microservices
- Data Lake, DWH
- Constructing Data Lakes for storing collected data
- Managing partial permissions and resources on DWH
- Data utilization
- Developing tools to support use of data in Data Lakes and on DWH
- Developing a platform to provide stream data processing function
- Company-wide
- Improving performance, aiming for low latency/high throughput according to application and middleware development, operation, and requirements
- Developing tools to automate operations and/or lower costs
- Identifying and solving technical issues with the system caused by engineering
- Tools Used/Dev Environment
- Programming languages: Python, Java, Scala
- Infrastructure: Google Cloud Platform (Compute Engine, Kubernetes Engine, Dataflow, Pub/Sub, Cloud Storage, BigQuery, Spanner, Cloud SQL, Cloud Logging etc.), Confluent Cloud
- Distributed processing: Apache Flink / Apache Beam(Dataflow) / Kafka Connect
- Workflow: Argo Workflows
- BI tools: Looker / Data Studio
- Configuration management tools: Terraform
- Monitoring: Stackdriver Monitoring / Datadog / PagerDuty
- CI/CD: Github Action / Spinnaker
- Knowledge Tool: Confluence/Jira
- Other: GitHub / Slack
- Development Method: Agile (Scrum, Kanban, etc.)
- Dev Organizations
- As part of the Data Engineeringamp; Machine Learning Team, you will work on development alongside members with a broad range of expertise from the following organizations.
- Teams
- Data amp; Machine Learning Platform
- Data Management Team
- Data Analytics Teams
- Machine Learning Teams (Search, Recommendation, Credit, Anti Fraud, etc)
- Product Teams (Marketing, CRM, Customer Support, Anti Fraud, etc)
- Machine Learning (Credit)
- Machine Learning (Anti-Fraud)
- Opportunity to build large-scale data infrastructure to support the massive amounts of data generated by Mercrai Group, including the Mercari Marketplace app with its over 20M monthly users.
- Work on development of an ecosystem to support data-driven business expansion by addressing the data utilization needs of many domains including analytics/decisi