LCI·GRAPH

An open-source pipeline that turns the LCA literature into a queryable inventory database.

The problem

Some of the best life-cycle inventory data in the field is published in academic papers — and then effectively buried. The numbers live in PDF tables and supplementary files, in whatever format the authors happened to use. If you need them, you re-transcribe them by hand; if you want to compare across studies, you first have to undo every paper's idiosyncrasies. Practitioners redo this work over and over, privately, and throw it away.

What it does

LCI·GRAPH ingests a paper and its supplements, extracts the inventory tables, links every flow to a canonical dataset, and stores the result in a database you can actually query — then exports to the formats practitioners already use (openLCA JSON-LD, SimaPro CSV, ILCD, EcoSpold2).

The whole machine, top to bottom. Each layer below gets its own close-up: extraction (Part 1), identity (Interlude), graph (Part 2).

LCI·GRAPH study browser: a list of ingested papers with DOI, system boundary, background database, and review status. — Papers in the database — each with its DOI, system boundary, and background database preserved.

Part 1 — the extraction pipeline

The pipeline's job is not just to find the numbers in a paper — it's to keep everything that makes those numbers meaningful. On the left: a table buried on page S12 of a supplementary PDF. On the right: the same inventory as typed, unit-checked rows in the database, every value traceable back to that table.

From paper to database

Table S6 from the paper's supplementary PDF: LCI data of CSTR 10 bar using serpentine, with columns for electricity, thermal energy, and carbon footprint per life-cycle stage. — Before: Table S6 in the supplementary PDF of Ostovari et al. 2020 — electricity, thermal energy, and footprint per life-cycle stage, with footnoted assumptions.

The same table after extraction into LCI·GRAPH: typed flows with amounts and units, methodology context, export buttons, and data-quality warnings. — After: the same table in LCI·GRAPH. The 20 / 324 / 441 kWh rows are all there — and where the paper is ambiguous (an empty thermal-energy cell), the pipeline raises an explicit data-quality warning instead of silently guessing. The footnotes survived too, as assumptions on the dataset.

What has to survive extraction

Amounts are data, not text to clean up. Every amount is stored as a verbatim string, so scientific notation survives — and an explicit zero ("PPS granulate — 0 g/kg") is kept as an author claim (we looked; it's nothing), not dropped as a missing value.

The reference flow is the denominator everything hangs on. Each process is stored normalized to one unit of its declared reference flow ("per kg of REM alloy"). When processes chain together, upstream inventories are rescaled by the exchange ratio at read time — the scaled version is never stored, because that's where conversion errors would live.

An inventory without its boundary conditions is not data, it's decoration. Every extracted dataset carries its methodology context — system boundary, LCIA method, allocation, source DOI — displayed next to the numbers wherever they appear. Two inventories are only comparable when you know what each one counted.

An extracted unit process: PEM electrolysis stack construction, with methodology context and a flow table mapped to ecoinvent datasets. — An extracted unit process: flows mapped to canonical ecoinvent datasets, methodology context pinned to the top.

The stack: LLMs draft, rules decide, humans approve

Pipeline diagram: input paper and supplements flow through a deterministic routing gate into multi-pass LLM extraction, then deterministic and LLM validation, then human review, then SQLite, then one-click exports. — The pipeline, end to end. Blue is where the model runs, grey is deterministic code, orange is the human. The model only ever drafts — the transitions between stages are all rules.

Exactly one of three extraction paths runs per paper, and routing is a file check, not a model call. Papers with supplements go supplement-first: every document is parsed to text with a table registry (pdfplumber, python-docx, openpyxl), a first pass reads the main paper for metadata and boundary context only, and a second extracts the LCI tables in one structured call. PDF-only papers go vision: the model reads the PDF pages directly — the corpus's gold standard, at 3–5× the cost. A plain text path is the last-resort fallback.

Nothing the model says enters the database as fact. Extractions land as pending review: deterministic checks run first (schema, inter-process links, variable amounts resolved against the table registry, ghost processes dropped), then a final LLM pass re-sends every table's JSON with its source table and asks "does this match?" Corrections are made in a review interface — and those corrections feed back as extraction rules, so the same mistake is not made twice.

Accuracy is measured, not asserted. The pipeline is evaluated against a golden set of hand-verified extractions, so a prompt change or model swap shows up as a number, not a vibe.

Meta-analysis view: canonical materials ranked by number of flows and studies they appear in. — Once flows are normalized to canonical materials, cross-study questions become answerable — e.g. which materials the literature actually covers.

Export to the tools you already use

A database nobody can get data out of is a roach motel. Every study and every individual process exports with one click to the formats practitioners actually work in: openLCA JSON-LD, SimaPro CSV, ILCD (GaBi), and EcoSpold2. The openLCA export is a standards-compliant JSON-LD package — processes, flows, units, and flow properties as linked entities:

{
  "@type": "Process",
  "name": "CSTR 10 bar using serpentine — state-of-the-art",
  "processType": "UNIT_PROCESS",
  "exchanges": [
    {
      "@type": "Exchange",
      "isInput": true,
      "amount": 324.0,
      "unit": { "name": "kWh/ton stored CO2" },
      "flow": { "name": "Electricity", "flowType": "PRODUCT_FLOW" }
    },
    …
  ]
}

In openLCA it's File → Import → JSON-LD, and the process drops into the navigation tree ready to build a product system from. No re-typing, no unit conversion errors, no lost footnotes. Here's the whole round trip:

Export side: from the study page to a downloaded openLCA JSON-LD package in one click.

Interlude — normalizing to canonical materials: how corrections become memory

Between extraction and the graph layer sits a less glamorous problem that decides whether any of the comparison features work at all: identity. "What substance is this exchange?" The naive answer — normalize the flow's name, and on a miss create a new material — is a fragmentation generator. After twelve studies the database held 616 materials, of which only 12 were shared across studies. One study's electricity was shattered across seven process-specific nodes ("Ball milling — electricity consumption", …) — 28 exchanges — while two other papers shared a single canonical Electricity. Cross-paper comparison is only as good as this join.

The fix rests on three choices. Read both channels of evidence: the free-text name and the linked background dataset — "Energy consumption during electrolysis" shares zero words with Electricity, but its dataset reference says market for electricity, medium voltage. Lookup, not retrieval: with an exact key the right answer is deterministic code, identical every time — embeddings have no concept of "exact hit, stop looking". Only genuinely novel strings earn an LLM, and there the model proposes candidates; code owns identity. Corrections become memory: every human decision is an event in an append-only log, and memory is rebuilt from it — aliases on materials, a dataset→material map. The channels teach each other: a flow matched via its dataset writes its name as an alias, so the next paper using that name matches even without a dataset. And on a genuine miss the flow is queued for review instead of minting a new material — the one change that stops fragmentation growing.

One subtlety: a dataset is not always an identity. Papers model substances with proxy datasets — one study's neodymium fluoride rides on a neodymium oxide dataset, deliberately. So identity follows the name channel; the dataset channel is evidence; and disagreement between the two is the proxy detector, routing the flow to a human with both votes shown.

The eval: simulate the memory before building the engine

With the decision log nearly empty (one human correction when this ran), a replay eval had nothing to replay. What could run before writing any engine code was a retrospective simulation: replay the corpus in ingest order, grow the two-channel memory as the engine would, and measure what it would have known — coverage potential, not correctness.

ingest-order replay of 1,039 exchanges, two-channel memory:
  53.2%  exact (name, dataset) pair replay — mostly repeats within a study
  10.3%  dataset channel — dataset string already seen in another paper
   5.2%  name channel — alias seen before
  31.3%  genuinely cold — no key fires; queue for a human

electricity gap spot check:
  all 28 orphaned exchanges share ONE normalized dataset key
  ("electricity, medium voltage") — the same key backing canonical Electricity.
  the dataset channel alone closes the entire enumerated gap.

Two-thirds of the corpus's exchanges were knowable in advance through channels the matcher never read. The caveats mattered too: junk dataset strings ("not specified") need a stoplist, a third of flows have no usable dataset reference at all, and one dataset can legitimately back different materials — so the dataset channel alone never gets to decide silently.

First contact with reality

Then the design met the corpus, in a first human review pass — and the corpus taught back. The pass forced the tooling into existence: pending flows pre-grouped into 425 decision buckets by shared dataset key, each with evidence-ranked canonical candidates. Twenty decisions later the enumerated electricity gap was closed — all 28 exchanges now point at the single canonical Electricity — and the review queue dropped from 1,148 to 1,013 links. Two lessons the design had missed, both now encoded:

Left — typography is an identity attack: three spellings of one substance had become three materials. Nineteen such duplicate groups hid in the corpus; the safe lenses (unicode-fold, curated synonyms) now auto-match at ingest, and a merge keeps every absorbed spelling as an alias — never asked twice. Right — an exchange's role is part of its identity: consumption (a burden you buy), disposal (a burden you pay for), and avoided product (a credit, the opposite sign) share a dataset key but are different claims — buckets split by exchange class, so they are never decided together.

The pipeline behind it. Grey is deterministic code, orange is the human, blue is the model — which proposes candidates on a genuine miss but never decides identity. Every correction flows through one write path into the decision log, and memory is rebuilt from the log.

Part 2 — the graph layer, put to a falsifiable test

The relational store works, but the data underneath it is graph-shaped: activities connected by flows, with identity ("what substance is this?"), provenance ("which paper says so?"), and hierarchy ("this part belongs to that product"). Every new comparison feature kept dying on the relational shape. Still, "use a graph database" is also a fashionable answer — so instead of migrating on faith, I framed it as a falsifiable hypothesis and built a proof of concept that was designed to be able to fail.

The hypothesis: a labeled-property-graph vocabulary of five node labels (Activity, Exchange, Material, Dataset, Source) and eight edge types can hold both paper-extracted inventories and an industrial bill of materials without schema additions, and turns the comparison and composition questions into simple traversals that are painful over the relational equivalent. Four sub-claims made it falsifiable:

H1 — the vocabulary is sufficient. Both worlds represented faithfully, or a sixth node label / ninth edge type has to appear.

H2 — identity is one problem. Cross-paper matching ("same electricity, three papers") and intra-study linkage are both missing edges of the same type, closable by one engine — falsified if either world needs special-casing.

H3 — roll-up works. A study's internal flows cancel and its boundary exchanges sum to a correct black-box view, with chain math computed at read time — falsified by disagreeing with a hand-calculated roll-up.

H4 — the graph pulls its weight. The test questions take materially less effort than their SQL equivalents — falsified, honestly, if they don't.

The vocabulary

The heart of the vocabulary is bipartite — Activities (things that happen) and Exchanges (quantified stuff crossing their boundary). Exchange is a first-class node for a structural reason: in a property graph, edges can't have edges — and provenance, dataset mapping, and canonical identity all need something to attach to. Around that heart: a Material library for identity, Dataset for background references, and Source so every node knows where it came from.

Graph schema diagram: Activity and Exchange as the bipartite heart, connected by PRODUCES, CONSUMES, and FEEDS edges; Exchange points to Material via INSTANCE_OF, to Dataset via FROM_DATASET, and everything points to Source via SOURCED_FROM. — The v0 vocabulary. Grounded in two published approaches (LCIKG, GraphDLCA) and one decision that shaped everything: Exchange-as-node.

A real paper in the vocabulary

This is an actual extraction redrawn in the vocabulary — a study on NdFeB magnet recycling, mapped from the live database. Two design surfaces show up immediately. The provider's output and the consumer's input are the same stuff under different strings ("milled magnet powder (ca 200 µm)" vs "cleaned, demagnetized, ball-milled EOL magnet") — so intra-study linkage and cross-paper identity are one matching problem, not two. And the 6.45 kg on the chain link is where the denominators change: upstream amounts are per kg of powder, downstream per kg of alloy, so roll-up math has to multiply through the conversion at read time.

Instance diagram of the magnet recycling study: pre-treatment and electrolysis activities connected through exchanges, with the FEEDS chain link carrying the 6.45 kg conversion, an electricity exchange pointing to the canonical Electricity material and its background dataset, variant edges, and provenance edges to the source paper. — Study 34 in the graph: two unit processes, their boundary exchanges, the chain link, the canonical-material join, and the scenario-variant edge — all from one supplementary table, all traceable.

The test setup

The architecture is a read-only projection, not a migration. SQLite stays the system of record — the LLM extraction pipeline, the evaluation harness, and the web app never touch the graph. A mechanical exporter rebuilds a Neo4j projection from it, and if the hypothesis failed I'd delete the projection and nothing was lost:

papers / ERP ─► extraction pipeline ─► SQLite (system of record)
                                             │  exporter: 1:1, full rebuild
                                             ▼
                                       Neo4j 5 (Docker)
                                             │
                                             ▼
                              Cypher checklist + hand-check scripts

Two decisions made the exporter boring, which was the point. First: exchanges are first-class nodes (Exchange-as-node), so every row type in the relational schema has exactly one graph counterpart — the mapping is 1:1 and the exporter is ≈180 lines with no judgment calls. Second: full rebuild, always — no incremental sync, no migration logic, the projection is disposable by design. The engine is Neo4j 5 in Docker, the production target in the specs and the engine both grounding papers used; an embedded Cypher engine was the named fallback if infra friction appeared — the vocabulary and the queries wouldn't change.

The test was a six-question checklist run on real data — three papers on rare-earth magnet recycling, plus an industrial BoM (an ERP export for a polyester knit fabric, three hierarchy levels deep). Two disciplines kept it honest. Every answer that could be checked by hand, was: the graph's numbers were diffed against hand-calculated results computed independently from SQLite and the ERP spreadsheet. And the falsification clause was real: I wrote and ran SQL versions of the checklist side-by-side, because "SQL is equally easy" was a valid outcome.

Results

Neo4j graph view of an industrial bill of materials: three assembly levels (orange), twelve leaf exchanges (yellow), and eleven canonical materials, connected by PART_OF, CONSUMES, and INSTANCE_OF edges. — An industrial BoM living in the same graph as the paper world: three assembly levels, twelve leaf exchanges, eleven canonical materials. The disperse dye at the top is *one* node consumed by two process steps — in the spreadsheet it's two rows joined only by a human reading the article number.

The vocabulary held (H1). Both worlds — academic LCIs and an ERP-sourced BoM — were represented with zero new node labels and zero new edge types, and every node traces back to its source: 100% provenance completeness. Suppliers rode along as edge properties, and the SAP article numbers gave identity links for free — no fuzzy matching in the ERP world at all.

The identity gap became enumerable (H2). One study's electricity was fragmented across seven process-specific material nodes — 28 exchanges — while the other two papers shared a single canonical electricity node. In the graph that gap is one query's worth of missing identity edges, closable by a single matching engine with no per-study special-casing; in the relational store it was invisible except by eye.

The roll-up math was checked by hand (H3). Collapsing a study to one black box — internal flows cancel, boundary flows sum, quantities rescale along the chain — matched a hand-calculated roll-up exactly, for all eight scenario variants of the study, including the variants where the chain's conversion denominator changes mid-chain. Writing the query surfaced a real design rule: a strict black-box must cancel both sides of an internal link, or the provider's reference product leaks into the boundary.

The honest nuance (H4). On flat lookups, the graph and plain SQL tied — near-identical line counts, same results, recorded honestly. The divergence showed up where hierarchy lives: flattening the BoM — "a lot of work" when done by hand for every downstream consumer — is one depth-agnostic traversal (PART_OF*1.. with a running quantity product), reproducing the hand-flattened ERP export exactly, down to 1083.5188 kg of PET chips per tonne of fabric through two nested levels. The SQL equivalent is a recursive CTE carrying the same running product. So the graph earns its keep on composition, roll-ups, and provenance chains — and that nuance is load-bearing: the migration spec's acceptance criteria are anchored on those hierarchical queries, not on corpus-scale lookups where the graph can't justify itself.

The limits, recorded. One industrial BoM (n=1, textiles) with zero material overlap against the paper corpus — the cross-world join is machinery-tested, not data-tested. Multi-hop roll-up is untested, because every chain in the corpus is depth-1. And line-count is a weak proxy for effort. All three are written into the specs as open risks, not swept under the rug.

Where it stands

The gate verdict was GO: the hypothesis is confirmed — H4 in its nuanced form — and the migration specs were re-scoped on the evidence. The migration is now defined as the same full-rebuild exporter (the projection stays disposable, and match-decision history stays relational). The 28-exchange identity gap the PoC enumerated is meanwhile closed — by the human pass through the new review tooling, not yet by automation. Next is the unified matching engine, but its job has changed shape: not "close the gap" — make the next pass mostly automatic, by cashing in the coverage the simulation measured (two-thirds of exchanges knowable in advance) against the decision log the passes are now accumulating. Then the engine joins BoM materials to paper materials, and the production seam lets the graph backend replace the relational one behind a flag.

Screenshots are static snapshots of the local platform; there is no live demo — the database contains other people's published data, shown here for illustration.