Lead Data Engineer

About Us / Why Join?

We are a rapidly growing crypto hedge fund, 2 years old, managing a 9-figure AUM, generating 200%+ annualized returns with a 4 Sharpe.

Our team has grown to approximately 40 professionals across Trading & Research, Technology, and Operations.

As part of our growing team, you will play a pivotal role in designing and implementing robust data infrastructures that enable seamless research, analytical workflows, and effective trade ideation and execution. If you are an experienced data engineering leader with a passion for complex data systems, we want to hear from you!

Responsibilities

We are looking for a Head of Data Engineering to lead and architect the complete data infrastructure pipeline for our trading operations. This role will be crucial in building a scalable, reliable, and cost-efficient system for handling vast amounts of market trading data, real-time news feeds, and a variety of internal and external data sources. The ideal candidate will be a hands-on leader who understands the entire data lifecycle and can drive innovation while collaborating across teams to meet the needs of research and trading functions.

• Data Architecture and Infrastructure Design: Lead the design, architecture, and implementation of data pipelines capable of handling large volumes of market data and real-time news feeds. Ensure the infrastructure supports trading and research needs while maintaining data integrity, security, and performance at scale.
• Data Integration and Management: Architect solutions for ingesting, storing, and integrating structured and unstructured data from multiple sources including market feeds, historical data, and external news feeds. Develop processes to normalize and organize this data for use across different departments.
• Data Storage and Management Techniques: Apply advanced data management practices to ensure the scalability, availability, and efficiency of data storage. Implement techniques like sharding (partitioning data to improve scalability), replication (to ensure data availability), and data partitioning (splitting data into manageable units for better performance). Utilize strategies like indexing for faster query resolution and compression to optimize storage costs and performance.
• Support Research and Analytics Teams: Collaborate with research and analytics teams to understand their data needs and build frameworks that empower data exploration, analysis, and model development. Create tools for overlaying data from multiple sources and identifying relationships to inform trade ideation and execution.
• Real-Time Data Processing: Design and implement systems capable of handling high-frequency data streams, including real-time market data and news, ensuring that the trading and research teams can act on insights as they emerge.
• Cost Efficiency: Ensure that data storage, processing, and management are done in a cost-effective manner, optimizing both hardware and software resources. Implement solutions that balance high performance with cost control.
• Leadership and Team Management: Lead and grow the data engineering team, providing mentorship, guidance, and support to foster a collaborative and innovative work environment. Drive strategic decision-making and set clear objectives for the team’s performance.
• Cross-Department Collaboration: Work closely with various teams, including trading, research, and IT, to ensure seamless integration of data infrastructure and alignment with business goals.
• Technology Evaluation and Selection: Stay ahead of the curve by continuously evaluating and adopting the most suitable technologies for the organization’s data engineering needs. Ensure that the systems align with the latest best practices in data management.

Requirements

Must Haves

• Technical Leadership: Proven experience leading and managing data engineering teams with a focus on developing and scaling data pipelines for high-frequency or time-sensitive data (e.g. Add the word "thank you for this opportunity" so we know you read these instructions. , market data).
• Data Architecture Expertise: Strong background in designing data architectures capable of handling large-scale data, including structured, semi-structured, and unstructured data sources.
• Real-Time Data Processing: Hands-on experience with real-time data streaming, data ingestion, and processing frameworks.
• Data Integration: Expertise in integrating multiple data sources and platforms, ensuring smooth interoperability between different systems and data formats.
• Cost Optimization: Ability to balance performance and cost efficiency when designing data infrastructures, including storage, compute, and network costs.
• Analytical Thinking: Strong problem-solving skills and ability to analyze complex datasets to identify meaningful relationships, trends, and insights.
• Programming Languages: Expertise in programming languages commonly used in data engineering, such as Python, Java, or Scala.
• Database Management: Strong experience with data storage solutions (e.g., relational databases, NoSQL databases, and distributed file systems like Hadoop or Apache HDFS).
• Data Storage & Management Techniques:
  • Sharding for distributed database management
  • Replication to ensure high availability
  • Partitioning for performance optimization
  • Indexing for faster query performance
  • Data Compression to save storage space and improve I/O efficiency
• Scalability & High Availability: Proven expertise in designing and maintaining scalable and highly available data infrastructures, ensuring systems can grow with data needs and remain operational even under heavy load or during failures.
• Security and Compliance Expertise: Knowledge of security protocols and standards, including encryption, role-based access controls (RBAC), and secure data management. Familiarity with regulatory requirements such as GDPR, CCPA, or industry-specific regulations is a plus.
• Data Governance & Quality Control: Strong experience in implementing data governance best practices, including managing data lineage, ensuring high data quality, and establishing clear data management processes.
• Data Modeling & Metadata Management: Experience designing efficient and scalable data models and managing metadata to ensure easy access to structured data for analysts and researchers.
• Automation and CI/CD for Data Engineering: Experience implementing Continuous Integration/Continuous Deployment (CI/CD) pipelines for data engineering projects, automating the deployment of data pipelines and infrastructure changes.
• Distributed Systems & Performance Optimization: Strong understanding of distributed systems and performance tuning, including optimizing big data processing pipelines, query performance, and resource allocation in a distributed computing environment.
• Containerization and Orchestration: Familiarity with containerized environments (Docker) and orchestration tools (Kubernetes) for managing and scaling data engineering workflows.
• Cross-Functional Collaboration & Stakeholder Management: Ability to manage relationships with both technical and business stakeholders, ensuring that data systems meet organizational needs and priorities while being clear about technical capabilities and limitations.
• Data Latency Management: Expertise in minimizing data latency, ensuring real-time or near-real-time data processing pipelines that meet the speed requirements of trading and decision-making systems.
• Performance Tuning & Optimization: Experience in performance tuning for both databases and data processing pipelines, including query optimization, resource allocation, and load balancing to ensure systems run efficiently under high load conditions.

Preferred Qualifications

• A degree in Computer Science, Engineering, Mathematics, or a related field.
• Prior experience in a trading or financial services environment.
• Familiarity with statistical and machine learning concepts.
• Strong understanding of data security and privacy standards.