Member of Engineering (Evaluations) - Poolside

About Poolside

Poolside exists to build a world where AI will be the engine behind economically valuable work and scientific progress. In this decade, the world will create Artificial General Intelligence, and we are building the company to achieve this through applied research, engineering, infrastructure, and deployment at scale.

About Our Team

We are a remote-first team sitting across Europe and North America that comes together once a month in-person for 3 days and for longer offsites twice a year.

Our R&D and production teams combine research and engineering-oriented profiles. Everyone deeply cares about the quality of the systems we build and has a strong underlying knowledge of software development to enable faster development iterations.

About the Role

Design and implement the infrastructure and tooling used by poolside researchers and engineers.

Your Mission

To ensure our progress leads to meaningful improvements to our users.

Responsibilities

• Research and implementation of evaluations and benchmarks for both base models and instruction-following models.
• Collaborate with applied research and product teams to define meaningful metrics and evaluations capturing progress on real-world software development skills.
• Work collaboratively in a team: plan future steps, discuss, and communicate clearly with peers.

Skills & Experience

• LLM Experience: Strong understanding and intuition of Large Language Models and their limitations.
• Engineering Background: Strong programming skills (ideally across multiple languages including Python) and familiarity with the full software development lifecycle.
• Technical Environment: Experience with Linux, strong algorithmic skills, and the ability to critically evaluate code quality policies.
• Mindset: Good taste, a curious mindset, and a drive to use modern tools and continuously improve.

Benefits

• Fully remote work & flexible hours
• 37 days/year of vacation & holidays
• Health insurance allowance for you and dependents
• Company-provided equipment
• Wellbeing, always-be-learning, and home office allowances
• Frequent team get-togethers
• Diverse, inclusive, people-first culture