Lead Data Engineer
London, Greater London, England
£80k - 100k per year
Lead Data Engineer - for award winning technology company specialising in Machine Learning on the Limit Order Book
My client provides customers with a patented machine learning library of algorithms to help and guide their research on the limit order book data. The algorithms (MATLAB, R, Python) are unique to the limit order book data set, allowing scalability over petabytes of data. They have a fully-automated framework running highly complex multi-step data processing code around the clock, scaling on-demand using AWS.
They are on the lookout for a Lead Data Engineer to join their data team on a permanent basis and the plan would be to add three more junior staff to the team over the next twelve months. You'll have the opportunity to build and improve a world class data platform to shape the future of what's possible with their data.
Lead Data Engineer - role and responsibilities:
- Owning the packaging and build of the data processing code into a container
- When external resources are required, optimize use of these resources
- Use the AWS ELK stack to monitor logging. Understand what jobs have failed and why. Re-run jobs and /or liaise with the development team as required.
- Designing and building a Python feature extraction library for the data. Running this library in the DAG and persisting results into SQL.
- Minimize cost - schedule jobs taking into account predictable returns in the spot price
Lead Data Engineer - skillset:
- Ownership of grid computing systems for data pipeline processing, using technologies like Airflow, Condor, SGE, Docker, Mesos, Chronos, Celery, IBM Symphony etc
- Strong coding and scripting skills (eg Python, C++, Java) for solving automation and data transformation tasks
- Docker containers
- Packer to build machine images
- SQL relational databases, preferably PostgreSQL.
Salary: Up to £100,000 (dependent on experience)
If you think you'd make a good Lead Data Engineer and think you have the right skills to be in this role, please don't hesitate to get in touch by sending your most up to date CV to firstname.lastname@example.org or alternatively apply via this advert.
Please note, this client is not able to sponsor so you will need to have the right to work in the UK .
Posted 136 days ago