Show HN: Hamilton's UI – observability, lineage, and catalog for data pipelines

{
  "by": "elijahbenizzy",
  "descendants": 10,
  "id": 40235944,
  "kids": [
    40237169,
    40244558,
    40237525,
    40244753,
    40249414
  ],
  "score": 40,
  "text": "Hey HN – Stefan and Elijah here from DAGWorks (<a href=\"http:&#x2F;&#x2F;dagworks.io&#x2F;\">http:&#x2F;&#x2F;dagworks.io&#x2F;</a>, YC W23).<p>If you don’t remember us from our previous HN launch (<a href=\"https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=35056903\">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=35056903</a>), we’re the authors of Hamilton (<a href=\"https:&#x2F;&#x2F;github.com&#x2F;dagworks-inc&#x2F;hamilton\">https:&#x2F;&#x2F;github.com&#x2F;dagworks-inc&#x2F;hamilton</a>), an open-source library for building self-documenting, modular dataflows in python that works for data, ML, LLM pipelines, &amp; even web-workflows.<p>We’ve been developing this UI for a while and we’re excited to say we open-sourced it! It comes out of the box with the following capabilities, and only requires a single line code change to get:<p>1. Execution + metadata capture, e.g. automatic code profiling<p>2. Data&#x2F;artifact observability, e.g. summary statistics over dataframes, pydantic objects, etc...<p>3. Lineage &amp; provenance of data, e.g. quickly see what is upstream &amp; downstream of code&#x2F;data.<p>4. Asset&#x2F;transform catalog, e.g. search &amp; find if feature transforms&#x2F;metrics&#x2F;datasets&#x2F;models exist and where they’re used.<p>While the UI currently only self-populates for Hamilton dataflows, we’re looking to expand to other frameworks (we’d love your feedback!).<p>Check out the following video for an overview: <a href=\"https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=0VIVSeN7Ij8\" rel=\"nofollow\">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=0VIVSeN7Ij8</a>, as well as the documentation: <a href=\"https:&#x2F;&#x2F;hamilton.dagworks.io&#x2F;en&#x2F;latest&#x2F;concepts&#x2F;ui&#x2F;\">https:&#x2F;&#x2F;hamilton.dagworks.io&#x2F;en&#x2F;latest&#x2F;concepts&#x2F;ui&#x2F;</a>.<p>We’re looking for feedback&#x2F;adopters – feel free to reach out if you have any questions!",
  "time": 1714656159,
  "title": "Show HN: Hamilton's UI – observability, lineage, and catalog for data pipelines",
  "type": "story",
  "url": "https://github.com/DAGWorks-Inc/hamilton/tree/main/ui"
}

{
  "author": "apache",
  "date": null,
  "description": "Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does. - apache/…",
  "image": "https://opengraph.githubassets.com/b14adcecc042f503e5492ea4eab373d2440fb5c09acdb25c24d30c037f8cfddb/apache/hamilton",
  "logo": null,
  "publisher": "GitHub",
  "title": "hamilton/ui at main · apache/hamilton",
  "url": "https://github.com/apache/hamilton/tree/main/ui"
}

{
  "url": "https://github.com/apache/hamilton/tree/main/ui",
  "title": "hamilton/ui at main · apache/hamilton",
  "description": "Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does. - apache/...",
  "links": [
    "https://github.com/apache/hamilton/tree/main/ui",
    "https://github.com/DAGWorks-Inc/hamilton/tree/main/ui"
  ],
  "image": "https://opengraph.githubassets.com/b14adcecc042f503e5492ea4eab373d2440fb5c09acdb25c24d30c037f8cfddb/apache/hamilton",
  "content": "<div>\n  <div><ul><li><div><ul><li><div><p><span>AI CODE CREATION</span></p><ul><li><a target=\"_blank\" href=\"https://github.com/features/copilot\"><div><p><span>GitHub Copilot</span><span>Write better code with AI</span></p></div></a></li><li><a target=\"_blank\" href=\"https://github.com/features/spark\"><div><p><span>GitHub Spark</span><span>Build and deploy intelligent apps</span></p></div></a></li><li><a target=\"_blank\" href=\"https://github.com/features/models\"><div><p><span>GitHub Models</span><span>Manage and compare prompts</span></p></div></a></li><li><a target=\"_blank\" href=\"https://github.com/mcp\"><div><p><span>MCP Registry<sup>New</sup></span><span>Discover and integrate external tools</span></p></div></a></li></ul></div></li><li><div><p><span>DEVELOPER WORKFLOWS</span></p><ul><li><a target=\"_blank\" href=\"https://github.com/features/actions\"><div><p><span>Actions</span><span>Automate any workflow</span></p></div></a></li><li><a target=\"_blank\" href=\"https://github.com/features/codespaces\"><div><p><span>Codespaces</span><span>Instant dev environments</span></p></div></a></li><li><a target=\"_blank\" href=\"https://github.com/features/issues\"><div><p><span>Issues</span><span>Plan and track work</span></p></div></a></li><li><a target=\"_blank\" href=\"https://github.com/features/code-review\"><div><p><span>Code Review</span><span>Manage code changes</span></p></div></a></li></ul></div></li><li><div><p><span>APPLICATION SECURITY</span></p><ul><li><a target=\"_blank\" href=\"https://github.com/security/advanced-security\"><div><p><span>GitHub Advanced Security</span><span>Find and fix vulnerabilities</span></p></div></a></li><li><a target=\"_blank\" href=\"https://github.com/security/advanced-security/code-security\"><div><p><span>Code security</span><span>Secure your code as you build</span></p></div></a></li><li><a target=\"_blank\" href=\"https://github.com/security/advanced-security/secret-protection\"><div><p><span>Secret protection</span><span>Stop leaks before they start</span></p></div></a></li></ul></div></li><li><div><p><span>EXPLORE</span></p><ul><li><a target=\"_blank\" href=\"https://github.com/why-github\"><span>Why GitHub</span></a></li><li><a href=\"https://docs.github.com/\" target=\"_blank\"><span>Documentation</span></a></li><li><a href=\"https://github.blog/\" target=\"_blank\"><span>Blog</span></a></li><li><a href=\"https://github.blog/changelog\" target=\"_blank\"><span>Changelog</span></a></li><li><a target=\"_blank\" href=\"https://github.com/marketplace\"><span>Marketplace</span></a></li></ul></div></li></ul><p><a target=\"_blank\" href=\"https://github.com/features\"><span>View all features</span></a></p></div></li><li><div><ul><li><div><p><span>BY COMPANY SIZE</span></p><ul><li><a target=\"_blank\" href=\"https://github.com/enterprise\"><span>Enterprises</span></a></li><li><a target=\"_blank\" href=\"https://github.com/team\"><span>Small and medium teams</span></a></li><li><a target=\"_blank\" href=\"https://github.com/enterprise/startups\"><span>Startups</span></a></li><li><a target=\"_blank\" href=\"https://github.com/solutions/industry/nonprofits\"><span>Nonprofits</span></a></li></ul></div></li><li><div><p><span>BY USE CASE</span></p><ul><li><a target=\"_blank\" href=\"https://github.com/solutions/use-case/app-modernization\"><span>App Modernization</span></a></li><li><a target=\"_blank\" href=\"https://github.com/solutions/use-case/devsecops\"><span>DevSecOps</span></a></li><li><a target=\"_blank\" href=\"https://github.com/solutions/use-case/devops\"><span>DevOps</span></a></li><li><a target=\"_blank\" href=\"https://github.com/solutions/use-case/ci-cd\"><span>CI/CD</span></a></li><li><a target=\"_blank\" href=\"https://github.com/solutions/use-case\"><span>View all use cases</span></a></li></ul></div></li><li><div><p><span>BY INDUSTRY</span></p><ul><li><a target=\"_blank\" href=\"https://github.com/solutions/industry/healthcare\"><span>Healthcare</span></a></li><li><a target=\"_blank\" href=\"https://github.com/solutions/industry/financial-services\"><span>Financial services</span></a></li><li><a target=\"_blank\" href=\"https://github.com/solutions/industry/manufacturing\"><span>Manufacturing</span></a></li><li><a target=\"_blank\" href=\"https://github.com/solutions/industry/government\"><span>Government</span></a></li><li><a target=\"_blank\" href=\"https://github.com/solutions/industry\"><span>View all industries</span></a></li></ul></div></li></ul><p><a target=\"_blank\" href=\"https://github.com/solutions\"><span>View all solutions</span></a></p></div></li><li><div><ul><li><div><p><span>EXPLORE BY TOPIC</span></p><ul><li><a target=\"_blank\" href=\"https://github.com/resources/articles?topic=ai\"><span>AI</span></a></li><li><a target=\"_blank\" href=\"https://github.com/resources/articles?topic=software-development\"><span>Software Development</span></a></li><li><a target=\"_blank\" href=\"https://github.com/resources/articles?topic=devops\"><span>DevOps</span></a></li><li><a target=\"_blank\" href=\"https://github.com/resources/articles?topic=security\"><span>Security</span></a></li><li><a target=\"_blank\" href=\"https://github.com/resources/articles\"><span>View all topics</span></a></li></ul></div></li><li><div><p><span>EXPLORE BY TYPE</span></p><ul><li><a target=\"_blank\" href=\"https://github.com/customer-stories\"><span>Customer stories</span></a></li><li><a target=\"_blank\" href=\"https://github.com/resources/events\"><span>Events &amp; webinars</span></a></li><li><a target=\"_blank\" href=\"https://github.com/resources/whitepapers\"><span>Ebooks &amp; reports</span></a></li><li><a target=\"_blank\" href=\"https://github.com/solutions/executive-insights\"><span>Business insights</span></a></li><li><a href=\"https://skills.github.com/\" target=\"_blank\"><span>GitHub Skills</span></a></li></ul></div></li><li><div><p><span>SUPPORT &amp; SERVICES</span></p><ul><li><a href=\"https://docs.github.com/\" target=\"_blank\"><span>Documentation</span></a></li><li><a href=\"https://support.github.com/\" target=\"_blank\"><span>Customer support</span></a></li><li><a target=\"_blank\" href=\"https://github.com/orgs/community/discussions\"><span>Community forum</span></a></li><li><a target=\"_blank\" href=\"https://github.com/trust-center\"><span>Trust center</span></a></li><li><a target=\"_blank\" href=\"https://github.com/partners\"><span>Partners</span></a></li></ul></div></li></ul></div></li><li><div><ul><li><div><p><span>COMMUNITY</span></p><ul><li><a target=\"_blank\" href=\"https://github.com/sponsors\"><div><p><span>GitHub Sponsors</span><span>Fund open source developers</span></p></div></a></li></ul></div></li><li><div><p><span>PROGRAMS</span></p><ul><li><a href=\"https://securitylab.github.com/\" target=\"_blank\"><span>Security Lab</span></a></li><li><a href=\"https://maintainers.github.com/\" target=\"_blank\"><span>Maintainer Community</span></a></li><li><a target=\"_blank\" href=\"https://github.com/accelerator\"><span>Accelerator</span></a></li><li><a href=\"https://archiveprogram.github.com/\" target=\"_blank\"><span>Archive Program</span></a></li></ul></div></li><li><div><p><span>REPOSITORIES</span></p><ul><li><a target=\"_blank\" href=\"https://github.com/topics\"><span>Topics</span></a></li><li><a target=\"_blank\" href=\"https://github.com/trending\"><span>Trending</span></a></li><li><a target=\"_blank\" href=\"https://github.com/collections\"><span>Collections</span></a></li></ul></div></li></ul></div></li><li><div><ul><li><div><p><span>ENTERPRISE SOLUTIONS</span></p><ul><li><a target=\"_blank\" href=\"https://github.com/enterprise\"><div><p><span>Enterprise platform</span><span>AI-powered developer platform</span></p></div></a></li></ul></div></li><li><div><p><span>AVAILABLE ADD-ONS</span></p><ul><li><a target=\"_blank\" href=\"https://github.com/security/advanced-security\"><div><p><span>GitHub Advanced Security</span><span>Enterprise-grade security features</span></p></div></a></li><li><a target=\"_blank\" href=\"https://github.com/features/copilot/copilot-business\"><div><p><span>Copilot for Business</span><span>Enterprise-grade AI features</span></p></div></a></li><li><a target=\"_blank\" href=\"https://github.com/premium-support\"><div><p><span>Premium Support</span><span>Enterprise-grade 24/7 support</span></p></div></a></li></ul></div></li></ul></div></li><li><a target=\"_blank\" href=\"https://github.com/pricing\"><span>Pricing</span></a></li></ul></div>\n        <div>\n  <div>\n    <div>\n    <p>\n      </p><h2 id=\"feedback-dialog-title\">\n        Provide feedback\n      </h2>\n    <p></p>\n  </div>\n        <div>        \n          <p>We read every piece of feedback, and take your input very seriously.</p>\n          <label>Include my email address so I can be contacted</label>\n</div>\n    <div>\n    <p>\n      </p><h2 id=\"custom-scopes-dialog-title\">\n        Saved searches\n      </h2>\n        <h2 id=\"custom-scopes-dialog-description\">Use saved searches to filter your results more quickly</h2>\n    <p></p>\n  </div>\n        <div>        \n        <div>\n          <div>\n            <p><label>Name</label></p>\n          </div>\n          <p><label>Query</label>\n          </p>\n          <p>\n            To see all available qualifiers, see our <a target=\"_blank\" href=\"https://docs.github.com/search-github/github-code-search/understanding-github-code-search-syntax\">documentation</a>.\n          </p>\n        </div>\n</div>\n  </div>\n            <div>\n              <p><a target=\"_blank\" href=\"https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2Fapache%2Fhamilton%2Ftree%2Fmain%2Fui\">\n                Sign in\n              </a>\n            </p></div>\n              <p><a target=\"_blank\" href=\"https://github.com/signup?ref_cta=Sign+up&amp;ref_loc=header+logged+out&amp;ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Ffiles%2Fdisambiguate&amp;source=header-repo&amp;source_repo=apache%2Fhamilton\">\n                Sign up\n              </a></p><div>\n      Appearance settings\n  </div>\n          </div>\n      </div>",
  "author": "",
  "favicon": "https://github.githubassets.com/favicons/favicon.svg",
  "source": "github.com",
  "published": "",
  "ttr": 33,
  "type": "object"
}