Bloomberg's internal and enterprise compute and data science platforms were established to support development efforts around data-driven compute, machine learning, and business analytics. Both the Data Science Platform and BQuant are solutions that aim to provide scalable compute, specialized hardware and first-class support for a variety of workloads such as Spark, PyTorch and Jupyter. These solutions are built using containerization, container orchestration and cloud architecture.
As the needs of distributed compute, machine learning, data exploration and analysis advance, so do the needs of the compute solution that underscores it. Accentuated by the widespread success of Large-Language-Models and AI initiatives across Bloomberg, these platforms are poised for continued growth to accommodate the endless number of products across Bloomberg that rely on a robust compute environment. Highlights from our upcoming roadmap focus on creating a highly scaled and performant compute solution that abstracts away common requirements that appear across many use cases, including creating a highly available federation layer for Batch Spark Workloads, increasing compute resource usage efficiency and visibility, enhancing the Interactive Spark experience, and continuing to enhance our cloud integration within BQuant's infrastructure.
As a member of the Spark Engineering Team, you'll have the opportunity to make key technical decisions to keep these solutions moving forward. Our team makes extensive use of open source (e.g. Spark, Kubernetes, Istio, Calico, Buildpacks, Kubeflow, Jupyter etc.) and is deeply involved in a number of communities. We collaborate widely with the industry, contribute back to the open source projects, and even present at conferences. While working on the platform, the backbone for many of Bloomberg's up and coming products, you will have the opportunity to collaborate with engineers across the company and learn about the technology that delivers products from the news to financial instruments. If you are a software engineer who is passionate about building resilient, highly available infrastructure and seamless, usable full stack solutions, we'd like to talk to you about an opening on our team.
We'll trust you to:
Interact with data engineers and ML experts across the company to assess their development flow and scale requirements
Solve complex problems such as cluster federation, compute resource management and public cloud integration.
Build first-class observability in a cloud-native way that provide insights that our users need
Educate users through tech talks, professional training, and documentation
Collaborate across data science teams on proper use/integration of our platform
Tinker at a low level and communicate your work at a high level
Research, architect and drive complex technical solutions, consisting of multiple technologies
Mentor junior engineers and be a strong engineering voice who takes charge driving part of Spark's technical vision
You'll need to have:
4+ years of programming experience with at least 2 object-oriented programming languages (Go, Python, Java) and willingness to learn more as needed
A degree in Computer Science, Engineering or similar field of study or equivalent work experience.
Experience building and scaling container-based systems using Kubernetes
Experience with distributed data analytics frameworks eg. Spark, Trino, Presto, Kafka
Ability to keep up with open source tech and trends for data analytics
A passion for providing reliable and scalable enterprise-wide infrastructure
We'd love to see:
Experience with Kubebuilder and Kubernetes operator-based frameworks
Experience working with platform security standards such as Spiffe and Spire
Experience with mainstream machine learning frameworks such as PyTorch, Tensorflow
Open source involvement such as a well-curated blog, accepted contribution, or community presence
Experience operating production systems in the public cloud e.g. AWS, GCP, or Azure
Experience with configuration management systems (e.g. Babka)
Experience with continuous integration tools and technologies (Jenkins, Git, Chat-ops)
If this sounds like you, apply! You can also learn more about our work using the links below:
Managing Multi-Cloud Apache Spark on Kubernetes https://www.youtube.com/watch?v=ZfphfcUJruI
Scaling Spark on Kubernetes - https://www.youtube.com/watch?v=GbpMOaSlMJ4
Kubeflow for Machine Learning: https://learning.oreilly.com/library/view/kubeflow-for-machine/9781492050117/
HDFS on Kubernetes: Tech deep dive on locality and security: https://conferences.oreilly.com/strata/strata-ca-2018/public/schedule/detail/63855
Bay Area Spark Meetup 2018: https://www.meetup.com/spark-users/events/250221273/
Apache Spark on k8s and HDFS Security: https://databricks.com/session/apache-spark-on-k8s-and-hdfs-security
Machine Learning the Kubernetes Way - https://www.youtube.com/watch?v=ncED2EMcxZ8
Inference with KFServing - https://www.youtube.com/watch?v=saMkA4fIOH8
ML at Bloomberg - https://on-demand-gtc.gputechconf.com/gtcnew/sessionview.php?sessionName=s9810-machine+learning+%40+bloomberg%3a+building+on+kubernetes
Kubernetes on Bare Metal - https://www.youtube.com/watch?v=svyuBSsMtxs
Serverless Inferencing on Kubernetes - https://arxiv.org/pdf/2007.07366.pdf
Serverless ML Inference https://www.youtube.com/watch?v=HlKOOgY5OyA
Bloomberg is an equal opportunity employer, and we value diversity at our company. We do not discriminate on the basis of age, ancestry, color, gender identity or expression, genetic predisposition or carrier status, marital status, national or ethnic origin, race, religion or belief, sex, sexual orientation, sexual and other reproductive health decisions, parental or caring status, physical or mental disability, pregnancy or maternity/parental leave, protected veteran status, status as a victim of domestic violence, or any other classification protected by applicable law.
Bloomberg provides reasonable adjustment/accommodation to qualified individuals with disabilities. Please tell us if you require a reasonable adjustment/accommodation to apply for a job or to perform your job. Examples of reasonable adjustment/accommodation include but are not limited to making a change to the application process or work procedures, providing documents in an alternate format, using a sign language interpreter, or using specialized equipment. If you would prefer to discuss this confidentially, please email AMER_recruit@bloomberg.net (Americas), EMEA_recruit@bloomberg.net (Europe, the Middle East and Africa), or APAC_recruit@bloomberg.net (Asia-Pacific), based on the region you are submitting an application for.
Salary Range: 160,000 - 240,000 USD Annually + Benefits + Bonus The referenced salary range is based on the Company's good faith belief at the time of posting. Actual compensation may vary based on factors such as geographic location, work experience, market conditions, education/training and skill level. We offer one of the most comprehensive and generous benefits plans available and offer a range of total rewards that may include merit increases, incentive compensation [Exempt roles only], paid holidays, paid time off, medical, dental, vision, short and long term disability benefits, 401(k) +match, life insurance, and various wellness programs, among others. The Company does not provide benefits directly to contingent workers/contractors and interns.
