Skip to main content

We have over 50,000 UK jobs available, so why not start a new job search on Zoek

Lead Data Engineer

  • Location: Nationwide / Work from Home
  • Salary: £80k - 100k per year
  • Contract hours: FullTime

Job provided by X4 Group

Share this job on social:

About the job

Lead Data Engineer - for award winning technology company specialising in Machine Learning on the Limit Order Book

My client provides customers with a patented machine learning library of algorithms to help and guide their research on the limit order book data. The algorithms (MATLAB, R, Python) are unique to the limit order book data set, allowing scalability over petabytes of data. They have a fully-automated framework running highly complex multi-step data processing code around the clock, scaling on-demand using AWS.

They are on the lookout for a Lead Data Engineer to join their data team on a permanent basis and the plan would be to add three more junior staff to the team over the next twelve months. You'll have the opportunity to build and improve a world class data platform to shape the future of what's possible with their data.

Lead Data Engineer - role and responsibilities:

  • Owning the packaging and build of the data processing code into a container
  • When external resources are required, optimize use of these resources
  • Use the AWS ELK stack to monitor logging. Understand what jobs have failed and why. Re-run jobs and /or liaise with the development team as required.
  • Designing and building a Python feature extraction library for the data. Running this library in the DAG and persisting results into SQL.
  • Minimize cost - schedule jobs taking into account predictable returns in the spot price

Lead Data Engineer - skillset:

  • Ownership of grid computing systems for data pipeline processing, using technologies like Airflow, Condor, SGE, Docker, Mesos, Chronos, Celery, IBM Symphony etc
  • Strong coding and scripting skills (eg Python, C++, Java) for solving automation and data transformation tasks
  • Docker containers
  • Packer to build machine images
  • SQL relational databases, preferably PostgreSQL.
  • Redis

Location: London

Salary: Up to £100,000 (dependent on experience)

Employment: Permanent

If you think you'd make a good Lead Data Engineer and think you have the right skills to be in this role, please don't hesitate to get in touch by sending your most up to date CV to #####@######.### or alternatively apply via this advert.

Please note, this client is not able to sponsor so you will need to have the right to work in the UK .



Share this job on social:


Before you go...

Did you know that Zoek has over 100,000 UK jobs available, your next job could be a click away! Start a new job search on Zoek!

Did you know that Zoek has over 100,000 UK jobs available, your next job could be a click away! Start a new job search on Zoek!