Show HN: Hamilton's UI – observability, lineage, and catalog for data pipelines

{
  "by": "elijahbenizzy",
  "descendants": 10,
  "id": 40235944,
  "kids": [
    40237169,
    40244558,
    40237525,
    40244753,
    40249414
  ],
  "score": 40,
  "text": "Hey HN – Stefan and Elijah here from DAGWorks (<a href=\"http:&#x2F;&#x2F;dagworks.io&#x2F;\">http:&#x2F;&#x2F;dagworks.io&#x2F;</a>, YC W23).<p>If you don’t remember us from our previous HN launch (<a href=\"https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=35056903\">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=35056903</a>), we’re the authors of Hamilton (<a href=\"https:&#x2F;&#x2F;github.com&#x2F;dagworks-inc&#x2F;hamilton\">https:&#x2F;&#x2F;github.com&#x2F;dagworks-inc&#x2F;hamilton</a>), an open-source library for building self-documenting, modular dataflows in python that works for data, ML, LLM pipelines, &amp; even web-workflows.<p>We’ve been developing this UI for a while and we’re excited to say we open-sourced it! It comes out of the box with the following capabilities, and only requires a single line code change to get:<p>1. Execution + metadata capture, e.g. automatic code profiling<p>2. Data&#x2F;artifact observability, e.g. summary statistics over dataframes, pydantic objects, etc...<p>3. Lineage &amp; provenance of data, e.g. quickly see what is upstream &amp; downstream of code&#x2F;data.<p>4. Asset&#x2F;transform catalog, e.g. search &amp; find if feature transforms&#x2F;metrics&#x2F;datasets&#x2F;models exist and where they’re used.<p>While the UI currently only self-populates for Hamilton dataflows, we’re looking to expand to other frameworks (we’d love your feedback!).<p>Check out the following video for an overview: <a href=\"https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=0VIVSeN7Ij8\" rel=\"nofollow\">https:&#x2F;&#x2F;www.youtube.com&#x2F;watch?v=0VIVSeN7Ij8</a>, as well as the documentation: <a href=\"https:&#x2F;&#x2F;hamilton.dagworks.io&#x2F;en&#x2F;latest&#x2F;concepts&#x2F;ui&#x2F;\">https:&#x2F;&#x2F;hamilton.dagworks.io&#x2F;en&#x2F;latest&#x2F;concepts&#x2F;ui&#x2F;</a>.<p>We’re looking for feedback&#x2F;adopters – feel free to reach out if you have any questions!",
  "time": 1714656159,
  "title": "Show HN: Hamilton's UI – observability, lineage, and catalog for data pipelines",
  "type": "story",
  "url": "https://github.com/DAGWorks-Inc/hamilton/tree/main/ui"
}

{
  "author": "asf-transfer",
  "date": null,
  "description": "Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does. - asf-transfer/h…",
  "image": "https://opengraph.githubassets.com/d7abac3c59d9f202e5ac6276b37c4cf79caf47556ae2fddbaf7dbc1acb9defe6/asf-transfer/hamilton",
  "logo": "https://logo.clearbit.com/github.com",
  "publisher": "GitHub",
  "title": "hamilton/ui at main · asf-transfer/hamilton",
  "url": "https://github.com/asf-transfer/hamilton/tree/main/ui"
}

{
  "url": "https://github.com/asf-transfer/hamilton/tree/main/ui",
  "title": "hamilton/ui at main · asf-transfer/hamilton",
  "description": "Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does. - asf-transfer/h...",
  "links": [
    "https://github.com/asf-transfer/hamilton/tree/main/ui",
    "https://github.com/DAGWorks-Inc/hamilton/tree/main/ui"
  ],
  "image": "https://opengraph.githubassets.com/d7abac3c59d9f202e5ac6276b37c4cf79caf47556ae2fddbaf7dbc1acb9defe6/asf-transfer/hamilton",
  "content": "<div>\n            <ul>\n                <li>\n      <div>\n                <ul>\n                    <li>\n  <a target=\"_blank\" href=\"https://github.com/features/copilot\">\n      <div>\n        <p>GitHub Copilot</p><p>\n        Write better code with AI\n      </p></div>\n</a></li>\n                    <li>\n  <a target=\"_blank\" href=\"https://github.com/security/advanced-security\">\n      <div>\n        <p>GitHub Advanced Security</p><p>\n        Find and fix vulnerabilities\n      </p></div>\n</a></li>\n                    <li>\n  <a target=\"_blank\" href=\"https://github.com/features/actions\">\n      <div>\n        <p>Actions</p><p>\n        Automate any workflow\n      </p></div>\n</a></li>\n                    <li>\n  <a target=\"_blank\" href=\"https://github.com/features/codespaces\">\n      <div>\n        <p>Codespaces</p><p>\n        Instant dev environments\n      </p></div>\n</a></li>\n                    <li>\n  <a target=\"_blank\" href=\"https://github.com/features/issues\">\n      <div>\n        <p>Issues</p><p>\n        Plan and track work\n      </p></div>\n</a></li>\n                    <li>\n  <a target=\"_blank\" href=\"https://github.com/features/code-review\">\n      <div>\n        <p>Code Review</p><p>\n        Manage code changes\n      </p></div>\n</a></li>\n                    <li>\n  <a target=\"_blank\" href=\"https://github.com/features/discussions\">\n      <div>\n        <p>Discussions</p><p>\n        Collaborate outside of code\n      </p></div>\n</a></li>\n                    <li>\n  <a target=\"_blank\" href=\"https://github.com/features/code-search\">\n      <div>\n        <p>Code Search</p><p>\n        Find more, search less\n      </p></div>\n</a></li>\n                </ul>\n              </div>\n</li>\n                <li>\n</li>\n                <li>\n      <div>\n                    <p><span>Explore</span></p><ul>\n                    <li>\n  <a target=\"_blank\" href=\"https://resources.github.com/learn/pathways\">\n      Learning Pathways\n</a></li>\n                    <li>\n  <a target=\"_blank\" href=\"https://resources.github.com/\">\n      Events &amp; Webinars\n</a></li>\n                    <li>\n  <a target=\"_blank\" href=\"https://github.com/resources/whitepapers\">\n      Ebooks &amp; Whitepapers\n</a></li>\n                    <li>\n  <a target=\"_blank\" href=\"https://github.com/customer-stories\">\n      Customer Stories\n</a></li>\n                    <li>\n  <a target=\"_blank\" href=\"https://partner.github.com/\">\n      Partners\n</a></li>\n                    <li>\n  <a target=\"_blank\" href=\"https://github.com/solutions/executive-insights\">\n      Executive Insights\n</a></li>\n                </ul>\n              </div>\n</li>\n                <li>\n      <div>\n              <div>\n                <ul>\n                    <li>\n  <a target=\"_blank\" href=\"https://github.com/sponsors\">\n      <div>\n        <p>GitHub Sponsors</p><p>\n        Fund open source developers\n      </p></div>\n</a></li>\n                </ul>\n              </div>\n              <div>\n                <ul>\n                    <li>\n  <a target=\"_blank\" href=\"https://github.com/readme\">\n      <div>\n        <p>The ReadME Project</p><p>\n        GitHub community articles\n      </p></div>\n</a></li>\n                </ul>\n              </div>\n          </div>\n</li>\n                <li>\n      <div>\n                <ul>\n                    <li>\n  <a target=\"_blank\" href=\"https://github.com/enterprise\">\n      <div>\n        <p>Enterprise platform</p><p>\n        AI-powered developer platform\n      </p></div>\n</a></li>\n                </ul>\n              </div>\n</li>\n                <li>\n    <a target=\"_blank\" href=\"https://github.com/pricing\">Pricing</a>\n</li>\n            </ul>\n        <div>\n  <div>\n    <div>\n    <p>\n      </p><h2 id=\"feedback-dialog-title\">\n        Provide feedback\n      </h2>\n    <p></p>\n  </div>\n    <div>\n    <p>\n      </p><h2 id=\"custom-scopes-dialog-title\">\n        Saved searches\n      </h2>\n        <h2 id=\"custom-scopes-dialog-description\">Use saved searches to filter your results more quickly</h2>\n    <p></p>\n  </div>\n  </div>\n              <p><a target=\"_blank\" href=\"https://github.com/signup?ref_cta=Sign+up&amp;ref_loc=header+logged+out&amp;ref_page=%2F%3Cuser-name%3E%2F%3Crepo-name%3E%2Ffiles%2Fdisambiguate&amp;source=header-repo&amp;source_repo=asf-transfer%2Fhamilton\">\n                Sign up\n              </a></p><div>\n      Appearance settings\n  </div>\n          </div>\n      </div>",
  "author": "",
  "favicon": "https://github.githubassets.com/favicons/favicon.svg",
  "source": "github.com",
  "published": "",
  "ttr": 19,
  "type": "object"
}