George Jieh

AI/ML Engineer San Francisco, CA

LLM Evaluation & RLHF for Frontier Models

Former Scale AI Oracle Tier trainer and Quality Analyst across twelve RLHF projects. Builds production LLM pipelines, ML systems, and multi-agent orchestration. Nine years of FINRA-regulated finance behind the judgment. Open to AI/ML engineering and research roles.

Explore selected work Read the story GitHub

01 / Story

Judgment built under pressure.

AI/ML engineer and LLM evaluation specialist in the Bay Area. Nine years in financial services under FINRA and SEC regulatory oversight before pivoting to machine learning. The finance career taught systems thinking under regulatory pressure. The AI work is where that thinking goes when the constraints are open-ended.

  • 12RLHF training projects
  • 9 yrsFINRA-regulated finance
  • 1stGoogle-sponsored capstone hackathon

At Scale AI / Outlier (August 2024 - April 2025), earned Oracle Tier and a Quality Analyst promotion across twelve RLHF training projects: process-supervised code reasoning, multi-turn instruction following, multimodal visual grounding, tool-use accuracy, and adversarial prompt design. Worked the data science, Python engineering, generalist reasoning, and finance domain tracks. Reviewed and corrected peer annotations, built evaluation rubrics from scratch, and caught reward hacking in quality review. Based on what each project trained for and the public improvements labs reported in the following months, this work plausibly fed into frontier models from OpenAI, Anthropic, Google DeepMind, Meta, and Microsoft.

Data science foundation from BrainStation (2023-2024) and the Google Data Analytics Professional Certificate. First place in the Google-sponsored capstone hackathon (Google Companion, April 2024): a geospatial pedestrian safety scoring model built from 1.9M+ Chicago crime records. Microsoft Certified Azure AI Engineer Associate and Azure Data Scientist Associate. PCEP certified in Python.

Currently rebuilding past projects to production quality while shipping new AI systems. Runs multi-agent orchestration across OpenAI Codex and Hermes Agent for coding workflows, knowledge management, and automated operations.

FINRA licensed (Series 7, 66, SIE). WSET Level 2 wine certified. Bilingual English and Mandarin. UC Berkeley BA in American Studies.

02 / Focus

Models, systems, and consequences.

  1. 01

    LLM Evaluation & RLHF

    Multi-dimensional rubric-based evaluation, step-level process supervision, adversarial prompt design, reward hacking detection, and structured feedback writing for frontier model training pipelines.

    RLHF / Chain-of-Thought / DPO / Oracle Tier

  2. 02

    Machine Learning

    Supervised and unsupervised learning, classification on imbalanced datasets, geospatial modeling, AutoML frameworks, and systematic architecture evaluation with documented trade-offs.

    scikit-learn / PyTorch / H2O AutoML / Keras

  3. 03

    Production AI Pipelines

    LLM inference pipelines with structured output enforcement, dual-prompt architecture, model routing between hosted APIs and local inference, and production reliability safeguards.

    Python / FastAPI / OpenAI API / Ollama

  4. 04

    Agentic Orchestration

    Multi-agent coding workflows and knowledge management across OpenAI Codex and Hermes Agent. System prompt engineering, scheduled jobs, cross-agent delegation, and MCP server design.

    OpenAI Codex / Hermes Agent / MCP / Ollama Cloud

  5. 05

    Data Engineering

    Web scraping with anti-bot techniques, SQLite databases with schema migrations, ETL pipelines, and data cleaning with documented rationale for every decision.

    SQL / Selenium / Pandas / NumPy

  6. 06

    Finance Domain

    FINRA-licensed (Series 7, 66, SIE) with 9 years in wealth management operations, portfolio analysis, and regulatory compliance. Ground-truth domain expertise for AI evaluation in financial contexts.

    Series 7/66/SIE / Portfolio Analysis / FINRA/SEC

03 / Work

Selected systems, shown honestly.

Two shipped systems and one active rebuild, each labeled by its current state and open for a closer look.

01 Shipped MachineRead: Public Website Audit Public-readiness audit for institutional websites. Scores any public URL for AI visibility, agent accessibility, scrapability, and search-discovery readiness. The free Essentials tier exposes a 13-group / 56-point contract over HTTP and MCP, plus a portable Agent Skill package.
  • Python
  • FastAPI
  • Next.js
  • TypeScript
  • Pydantic
  • MCP

Live in production at machineread.ai, where the Essentials audit runs free with no account required. FastAPI backend with a 301-test suite, Next.js + TypeScript frontend, OpenAPI 3.1 schema, MCP server card, and a portable Agent Skill for agent-driven invocation. Type-checked with strict TypeScript, schema-validated with Pydantic, contract-pinned via OpenAPI, and machine-readable end to end. Anonymous, rate-limited, SSRF-protected, no API keys needed. The public source tree is released as the MachineRead-Preview repository, and the demo page here runs the same audit contract against a free-tier backend.

Open live demo View repository
02 Shipped REED: Real-time Equity and Economic Digest Self-hosted market news agent. Scheduled briefs written from curated RSS feeds by a single-turn LLM call, with a terminal-style React dashboard to read them in. Bring your own model key.
  • Python
  • FastAPI
  • Pydantic
  • React
  • TypeScript
  • RSS
  • OpenRouter
  • APScheduler
  • Financial News

Local-first by design: clone the repo, run the setup wizard, and an in-process scheduler fires four sessions per trading day plus a Monday weekend recap, skipping days the NYSE is closed. The research step is deliberately kept off the model. Curated RSS feeds are fetched, deduplicated, and time-filtered first, then one LLM call with zero tools exposed turns those headlines into a structured digest with sentiment, tickers, and numbered sources. Stories whose URL is not in the pre-fetched link set are dropped, which is what prevents the model from inventing sources. An earlier build used a crawler and scraper; it was slow and frequently blocked, so the RSS pre-flight replaced it. Five provider classes are supported (Anthropic, OpenAI, OpenRouter, Ollama, any OpenAI-compatible endpoint) with no default model. The dashboard reads the local API in dev, or a public dataset repo in static-demo mode.

Visit project View repository
03 In progress Wine Grape Climate Suitability Model End-to-end ML pipeline predicting wine grape varietal suitability from regional climate data. Processed 143,000+ wine records and 60+ years of weather data (1961-2016), with a working Flask prediction interface.
  • Python
  • H2O AutoML
  • scikit-learn
  • Flask
  • Selenium
  • SMOTE

Built a custom Selenium scraper to supplement Kaggle datasets with Wine Enthusiast ratings. Engineered seasonal climate features aligned with vine growth cycles: budburst, flowering, veraison, harvest. Reached 56% accuracy with H2O AutoML Distributed Random Forest on a 19-class imbalanced dataset, roughly 10x the random baseline, and documented the data quality limits that cap it. Currently being rebuilt: production-quality modular codebase, better data sourcing and cleaning for a more reliable model, and a Gradio interface.

View repository
View all repositories on GitHub

04 / Contact

Let's talk about the work.

Open to AI/ML engineering and research roles. Also available for freelance AI training, evaluation, and annotation work on weekends.