Product engineering
From a Museum CSV to Searchable Data: A Lesson for Sheet Parser
By Vivek Jalondhara · 5 min read
Published
Use Simon Willison’s museum-data example to plan a small, useful vendor catalogue prototype with clear source relationships.
In May 2018, Simon Willison described turning the Metropolitan Museum of Art's public CSV into a searchable Datasette instance. The source listed 464,360 collection items. His short account offers a concrete example of giving existing tabular data a more useful interface.
The exercise below applies that idea to a fictional vendor catalogue. It is an original prototype plan inspired by the documented example, not an account of how Datasette was built or a claim that Sheet Parser uses its technology.
Start with a question the file cannot answer conveniently
A vendor sheet contains information, but finding the right item may still involve opening several files and remembering which version belongs to which supplier. A useful first question is modest: which uploaded source contains this exact product code? That question suggests a searchable catalogue before it suggests an autonomous agent.
Write the expected answer on paper. It should include the matching item, vendor, source version, and a way to inspect the original row. A result that only returns a product description leaves the user with the same reconciliation work after the search has finished.
Keep the first collection intentionally small
For a practice project, create a few fictional rows from two vendors. Include a shared product code to expose why the vendor matters, and preserve the code as text so leading zeroes survive. Give each source a stable identity rather than using whichever uploaded filename happens to look familiar.
A small fixture lets you inspect every transformation. If the first import loses a unit or merges two vendors, adding more documents only makes the problem harder to understand. The milestone is a correct answer to the chosen question, with enough source context to explain it.
Build a complete path through the data
The Python example below reads a tiny CSV, stores its text fields in an in-memory SQLite database, and performs a parameterised lookup. It demonstrates a complete input-to-answer path without a web interface or external service. Python's sqlite3 module provides the connection and bound query parameters used here.
The example intentionally stops before prices, authentication, and multi-user editing. For a portfolio walkthrough, explain that scope directly. A reader can understand what runs, inspect its output, and distinguish a working data transformation from the additional product work required to expose it safely to other people.
Add interpretation where the source needs it
Some vendor files have dependable column headers and can use a configured mapping. Others need help interpreting merged headings, descriptions, or inconsistent labels. Introduce an LLM only where there is a specific ambiguity to resolve, while preserving the same downstream draft contract.
This separates two experiments: whether the catalogue answers a useful question, and whether a model improves extraction from particular formats. Combining them immediately makes a failed demonstration hard to diagnose. A wrong search result might come from extraction, identity matching, storage, or the query itself.
Show the transformation as portfolio evidence
A useful demonstration can show one synthetic file, the resulting records, and a search that returns the original source. Add a short explanation of a deliberately difficult row and the decision made about it. These artefacts let a recruiter follow the engineering work without access to private vendor documents.
Keep claims proportional to the demonstration. A local lookup proves that the illustrated path works for its fixture. It does not establish production capacity, extraction accuracy, or business savings. Those questions need their own workloads and measurements if the prototype later becomes a maintained application.
Let the next user question choose the next feature
After exact lookup works, ask what makes the result actionable. A project team might need to select an accepted price revision, compare two suppliers, or see which rows still need review. Each answer has a different data requirement, so avoid treating a larger feature list as a stronger demonstration.
The motivating lesson is the usefulness of a finished connection between existing data and a question someone wants answered. A small prototype with inspectable inputs and outputs provides a concrete foundation for Sheet Parser's broader workflow: interpreting vendor sheets, maintaining shared records, and organising pricing by project.
A searchable catalogue from two fictional CSV rows
Run this educational Python example locally. It uses only synthetic data and an in-memory database. It does not create a public catalogue or process a real vendor file.
Illustrative example · Python · Simplified for explanation
import csv
import io
import sqlite3
source = "sku,description\nSPK-42,Ceiling speaker\nMIC-1,Desk microphone\n"
rows = list(csv.DictReader(io.StringIO(source)))
connection = sqlite3.connect(":memory:")
try:
connection.execute("CREATE TABLE items (sku TEXT, description TEXT)")
connection.executemany(
"INSERT INTO items VALUES (?, ?)",
[(row["sku"], row["description"]) for row in rows],
)
result = connection.execute(
"SELECT description FROM items WHERE sku = ?", ("SPK-42",)
).fetchall()
print(result) # [("Ceiling speaker",)]
finally:
connection.close()- Choose a useful lookup
- Prepare a synthetic CSV
- Import text without losing identity
- Return an inspectable result
Conceptual flow for the example above.
Tradeoff to keep in mind
An in-memory lookup is a teaching prototype. Durable storage, source links, vendor scoping, validation, and access control must be designed before using this pattern for shared operational data.