As a Deep Learning Data Engineer, your role will be instrumental in building and maintaining our data infrastructure, with a focus on handling large-scale data for training and inference of deep learning models. You will be responsible for data crawling, cleaning, and transforming raw data into formats suitable for training complex deep learning models. Your expertise in big data platforms like MapReduce, Hadoop, Spark, and Kubernetes will be crucial in efficiently processing and managing data for large deep learning tasks.

Job Responsibilities

• Data Crawling and Collection: Develop and implement advanced data crawling strategies to acquire vast amounts of structured and unstructured data from diverse sources, including websites, APIs, and databases.
• Data Cleaning and Preprocessing: Apply sophisticated data cleaning techniques to handle missing or inconsistent data, ensuring high-quality data for training large deep learning models.
• Data Transformation: Design and implement data transformation pipelines optimized for processing and preparing data for training complex deep learning models.
• Big Data Processing: Utilize your proficiency in big data platforms such as MapReduce, Hadoop, Spark, etc. to efficiently process and analyze large-scale datasets required for training large deep learning models.
• Database Management: Establish and manage databases tailored to store and access large volumes of processed data, ensuring data security, reliability, and efficient data retrieval.
• ETL (Extract, Transform, Load): Develop and maintain ETL workflows that effectively extract data from diverse sources, transform it to meet deep learning model requirements, and load it into data warehouses or databases.
• Performance Optimization: Optimize data processing workflows and algorithms to achieve superior performance for training and inference of large deep learning models.
• Data Modeling for Deep Learning: Collaborate closely with Data Scientists and Deep Learning Researchers to understand data requirements and design appropriate data models that cater to the needs of large and complex deep learning tasks.
• Data Governance: Implement robust data governance practices to ensure data accuracy, security, and compliance with data regulations, especially when working with sensitive data.
• Big Data Platform Management: Manage and configure big data platforms to ensure their stability, scalability, and seamless integration with deep learning workflows.
• Documentation: Document data engineering processes, data flows, and data models specific to large deep learning tasks, enabling knowledge sharing and future reference.