Register-based census: the number a state has to publish.
A national census asks one question and it sounds simple. How many people are there. Getting to an answer meant taking hundreds of millions of administrative records from thirteen government registers that had never agreed on anything, and reducing them to ten million records the state could stand behind in front of the United Nations Development Programme. Most of that work was not statistics. It was deciding which record to believe.
§ 01 — The problem
Kuwait ran its censuses the way most countries did. Population, buildings and dwellings, establishments: three exercises, once every five years, collected in the field on paper by enumerators recruited for the purpose.
The method worked, and it was expensive, slow, and increasingly beside the point. Policy cycles had stopped waiting five years. Planning, employment, housing and education decisions needed current numbers, and by the time a field census was compiled its picture was already historical. Kuwait also carried reporting obligations, national and international, including to the United Nations Development Programme, that a five-year cadence could not serve.
The alternative was known in principle. Every one of those residents already existed in government systems: civil registration, interior, education, justice, labour. The state was not short of data about its population. It had been accumulating it continuously for decades.
The problem was that no two systems agreed.
A register-based census replaces the field questionnaire with the records the state already holds. That is the entire idea, and it is one sentence. Everything difficult about it lives in the gap between "the state already holds this" and "the state can publish this."
§ 02 — Thirteen registers, none of which were built to be joined
The initial architecture integrated thirteen government entities. It scaled to seventeen in a second phase, against a target of twenty-one.
The Public Authority for Civil Information held civil registration. The Ministry of Interior held its own records. The Ministry of Education held schools.
The Ministry of Justice held the register nobody expects and everybody needs: marital status, divorce, property, legal lineage. A census does not count people. It counts households, and a household is a set of relationships rather than a set of individuals. Family statistics, buildings and housing, dwelling occupancy: none of those can be produced from a list of names. They require knowing who is married to whom, who was, who owns what, and who descends from whom. Justice is where a state keeps that. It is also the register that changes names, which feeds straight back into the reconciliation problem downstream.
Nine more followed.
Every one of those systems was correct on its own terms. Each had been built for its own mandate, by a different team, in a different decade, on a different platform. Relational databases, hierarchical databases, flat files, DB2, Sybase, mainframes. Different identifiers, different classifications, different record structures, different update cycles, different ideas about what a household was.
Some of the data did not arrive as a system at all. Hundreds of spreadsheets came in from individual schools, each one maintained by whoever maintained it, with the column order and the spelling conventions that person had chosen.
None of those thirteen registers had ever shared its data with another, and the data was classified at state level. It could not leave on a USB drive or a hard disk. Nothing could be extracted and carried somewhere convenient to be worked on. There was no staging copy on an analyst's workstation, and no export to a laptop for a weekend of profiling.
The obvious architecture for a problem like this one is to pull everything into a single place and clean it there. Under these conditions that architecture was not slow, and it was not expensive. It was not available. The processing had to go to the data, because the data could not go to the processing. Every technical decision described in the rest of this case study sits downstream of that sentence.
Oracle Streams moved three million rows a month into an Oracle 11g decision-support environment: replication over controlled links inside the government boundary rather than physical transfer of media, which was the only lawful shape the movement could take. The reporting layer was an internally developed J2EE application on Oracle Internet Application Server, with Cognos for the statistical outputs, running on Red Hat Enterprise Linux. Nine developers wrote the extract, transform and load logic: performance-oriented SQL, stored procedures and functions, because at this volume a query plan is an architectural decision rather than a tuning exercise.
That is the part of the project that is straightforward to describe. It was the smaller half of the work.
§ 03 — The records that rendered perfectly and matched wrongly
Arabic is where this project stopped being an integration exercise.
Three layers had to be made to agree before anything could be counted. Bytes, where one name is stored as different sequences of characters. Codes, where one short string means different things in different registers. Free text, where nothing is encoded at all and a clerk's description is the only signal there is. One problem standing at three levels: making things that mean the same thing actually be the same thing.
Bytes first.
The registers were in Arabic. They had been written over decades, onto systems with their own ideas about how Arabic should be stored, then exported through pipelines that had their own ideas about it again. What arrived was text that displayed correctly in every viewer anyone opened it in, and behaved incorrectly in the database.
Four encodings arrived across those registers. EBCDIC off the mainframes, and three the database named AR8MSWIN1256, AR8ASMO8X and AR8ISO8859P6. Those are the Oracle character set names for CP1256, ASMO 708 and ISO 8859-6, and they are what was actually configured, which matters because the configured name is the only place the assumption is written down. None of them announce themselves in a file. The worst pairing is the least obvious one. AR8MSWIN1256 is a superset of AR8ISO8859P6: the two agree across the core Arabic range and diverge in the positions ISO 8859-6 left empty, which CP1256 fills. Text written as one and read as the other therefore does not fail. Most of a name survives intact and one or two characters change. Too small to notice in a spot check, and more than enough to break a key.
That combination is the worst one available to a data architect. A visible defect gets fixed. An invisible defect gets argued about, because the person reporting it is contradicted by everyone else's screen.
The failure modes stacked.
Characters that are not there. Tatweel, the decorative elongation used to justify Arabic text, carries no meaning and breaks every comparison. Zero-width joiners and non-joiners. Bidirectional control marks inserted by export tooling. Non-breaking spaces and trailing whitespace. Presentation forms, where a character has been stored in its rendered shape rather than its canonical one. None of these are visible in any viewer. All of them are bytes, and bytes are what an index is built on. Indexes silently stopped being used, because a key with an invisible character in it is not the key that was looked up.
One name, spelled several legitimate ways. Arabic offers four alef forms, and a name may use any of them. Ta marbuta and ha are interchanged at word ends. Ya and alef maqsura vary by which convention a clerk learned. Diacritics appear in one register and not in another. Arabic-Indic and Western digits both occur in identifier and date fields. None of these are errors. Each register was internally consistent. They were consistent with each other in no way at all.
Text stored in the order it looked, not the order it meant. This was the worst of them. Some source systems stored Arabic in visual order, the sequence in which characters appear on a screen, rather than logical order, the sequence in which they are read and in which every piece of standard software expects them. Visually ordered text renders correctly. It sorts wrongly, joins wrongly, compares wrongly and indexes wrongly, and it does all of that without raising a single error anywhere in the stack. Two records holding the same name can carry entirely different byte sequences, and no query will ever bring them together.
Diagnosis was row-level work. Python, statistical profiling across the corpus, character-frequency analysis to find what had no business being there, and reconciliation logic that had to separate a genuine spelling variant from a genuinely different person. Where that work was permitted to run is the constraint asserting itself. Oracle external tables read the raw extracts where they already sat. External functions called Python from inside the database, so the repair executed within the boundary instead of on a copy taken outside it. Neither of those was a performance preference. They were the shapes the constraint left available. I found and repaired hundreds of distinct issue classes this way.
Underneath the encoding sat the harder problem. Once the bytes were right, the same human being still appeared under different spellings in different registers, with no shared key to join on. Arabic name reconciliation across independently maintained administrative registers is entity resolution without the identifier that makes entity resolution tractable. Getting it wrong in one direction invents residents. Getting it wrong in the other direction erases them. Both are census errors, and only one of them is visible.
§ 04 — The codes that matched perfectly, and the words that matched nothing
The Arabic problem was records that look identical and compare differently. Classification coding is the exact inverse: codes that compare cleanly and mean different things.
A census has to produce employment statistics, and the ones a state is asked for are cross-tabulated. Not how many people work, but how many people do a particular kind of work, in a particular kind of industry. Those are two different questions about the same working individual, and two different international classifications answer them.
Industry is answered by ISIC Rev. 4, the International Standard Industrial Classification of All Economic Activities, issued by the United Nations Statistics Division, adopted in 2006 and published in 2007. It classifies what an establishment does. It carries four levels: 21 sections, 88 divisions, 238 groups and 419 classes.
Occupation is answered by ISCO-08, the International Standard Classification of Occupations, issued by the International Labour Organization, adopted by a Tripartite Meeting of Experts in December 2007 and endorsed by the ILO Governing Body in March 2008. It classifies what a person does, grouped by skill level and skill specialisation. It also carries four levels: 10 major groups at one digit, 43 sub-major groups at two digits, 130 minor groups at three, and 436 unit groups at four.
Both were the current editions throughout this work. Each register that recorded anything about work did so in its own local scheme, built for its own operational purpose, at whatever depth that purpose required. Those registry codes were correct, current, and used every day. They were also local, and reporting to the United Nations Development Programme required the international standards. Local codes on their own would not be accepted, which meant that without a mapping there were no employment statistics at all.
So the mapping did not run from one scheme to one standard. It ran from several local schemes into two four-level hierarchies that answer different questions, and it had to know at every step which of the two it was serving.
The registers made that harder rather than easier. A field labelled something like "job" may hold the employer's economic activity, or the person's occupation, or an unstable mixture of the two depending on which clerk filled it and in which decade. Nothing in the data announces which. A value belonging in ISIC and a value belonging in ISCO sit in the same column, in the same format, and look exactly alike.
A local code and a standard code are both short strings. Two registers can hold the identical string and mean two different things, because each scheme assigned that string on its own authority. Two different strings can mean the same thing. String comparison, which fails loudly on Arabic names, succeeds silently here and gives the wrong answer.
The structure compounded all of it. Two target hierarchies, four levels each, and a mapping that had to run in both directions: from a detailed local code up to a section or a major group, and back down to a class or a unit group. The relationships were not one to one. Several local codes collapse into one standard code. One local code splits across several, depending on context held in a different field or in a different register entirely.
A lookup table assumes that one key resolves to one value. Neither half of that assumption survives contact with an occupational registry, and neither survives having two targets instead of one. What the mapping actually is, once you stop pretending otherwise, is a traversal across a graph: nodes at four levels of two hierarchies, edges that are many-to-one in one direction and one-to-many in the other, and a query that has to know which hierarchy it is walking, which level it entered at, and where it is permitted to come back down.
The third layer had no codes in it at all. Not every register recorded work as a code. In many records it was a description, typed by a clerk, in the words that person chose. This is the layer that forced the graph, and the case that forced it was Elementary Teacher.
A description carries information about two different levels of the hierarchy at once, and it carries them in different words.
The head noun locates the branch. Teacher puts the record in sub-major group 23, Teaching professionals. That is two digits. The output needs four, and the string has nothing further to give.
The qualifier selects the leaf. Elementary resolves to unit group 2341, Primary school teachers, inside minor group 234, Primary school and early childhood teachers, next to 2342, Early childhood educators. The branches it has to be kept out of are 233, Secondary education teachers, 231, University and higher education teachers, 232, Vocational education teachers, and 235, Other teaching professionals. Those are four distinct occupations that all answer to the word Teacher, and a census that cannot separate them cannot report on education staffing at all.
Neither fragment resolves on its own, and each one fails for a different reason.
The head noun needs the standard. ISCO-08 does not use the word elementary. It says primary. That is a fact about the classification rather than a fact about the text, and no quantity of string processing discovers it. The word carries a second hazard in the same classification: elementary is also the name of major group 9, Elementary occupations, which covers cleaners, labourers and helpers. Nothing in the string tells a matcher that the word means one thing standing beside Teacher and something unrelated standing beside occupations.
The qualifier needs the data. Many descriptions carry no qualifier, or carry one that could go either way. Teacher on its own leaves the leaf undecided, and the decision then has to come from elsewhere in the estate: the type of school in the employing register, an education-level field sitting in a different table, the establishment the person is attached to. The description is evidence. It is not always sufficient evidence.
So the algorithm does three things. It decomposes a description into the fragment that locates the branch and the fragment that selects the leaf. It routes each fragment to the level of the hierarchy where that fragment means something, which is not the same level for the two of them. And where the description runs out, it falls back to other register fields as evidence and resolves the leaf from those.
That is a traversal, and it is the reason the graph was necessary rather than convenient. You enter the hierarchy at whatever depth the description supports, then walk to the depth the output requires, and the entry depth is a property of the individual record rather than of the pipeline.
Classification mapping tooling that decomposed a description across two levels, routed the fragments into two different four-level hierarchies, and fell back to other registers as evidence where the words ran out did not exist to buy in 2010. I wrote it.
Bytes, codes, words. Three layers, one problem, and it is the problem the whole system existed to solve: making things that mean the same thing actually be the same thing, so that a state can count them once.
The three layers fail differently, and this one fails worst. An error in Arabic reconciliation corrupts a record, and a corrupted record can be found. An error in classification mapping corrupts nothing. It moves an entire population from one employment category into another, quietly, and the aggregate it produces looks perfectly reasonable to everyone who reads it. Nobody audits a plausible number.
§ 05 — One thousand three hundred rules that lived in people
The technical work would have produced nothing publishable on its own, because the question of what the data means had never been written down.
Thirteen hundred census rules decided how Kuwait's statistics were produced. Who counts as resident. How a household is constituted. What happens when two registers disagree about an address. When a record is included, excluded, or held. Those rules were real, they were consistently applied, and they existed in two places: in the books, and in the practice of a small number of very senior demographers who had been applying them for most of their careers.
Harvesting them meant sitting with the people who held them.
Those people were, for the most part, opposed to the digital transformation. That opposition was not irrational, and it is worth stating plainly rather than smoothing over. Their judgement was the quality control. They had watched systems arrive before. A machine that applied thirteen hundred rules without them would apply the rules that had been written down and not the ones nobody had ever had to articulate, and the second category is where thirty years of practice actually sits.
They were substantially right about that, which is why I extracted the rules as a designed methodology rather than a documentation exercise: conceptual, entity, physical and development data models; business process and activity models in BPMN and UML; entity relationship diagrams, XML schemas, decision matrices, requirements traceability, test plans and user acceptance materials. The point of that apparatus was not compliance. It was to make each rule visible enough that the demographer who held it could look at it and say whether it was right.
A rule nobody can see is a rule nobody can correct. That is the whole argument for writing them down, and it had to be made to the people with the most to lose from it.
§ 06 — Adjudication, and the number that got published
Hundreds of millions of raw records went in. Ten million came out.
That reduction is the project. It is not deduplication in the technical sense and it is not sampling. It is adjudication: for every person the state believes exists, deciding which of several conflicting administrative accounts is the one that gets published, and being able to say why.
Every one of those decisions had to satisfy three conditions at once. It had to be defensible to the demographers whose judgement the rules encoded. It had to be traceable back to the source record and the transformation that produced it. And the resulting figures had to meet United Nations Development Programme and national reporting requirements, which is an external standard rather than an internal one.
The resulting corpus supported the statistical outputs the census exists to produce: population and population trends, family statistics, industry and employment, buildings and housing, education.
Data quality measured 12 percent when the programme started and 99.5 percent at delivery. Those figures come from the project's own records rather than an external audit, and they are quoted here as what was reported rather than as what was independently verified.
The pilot succeeded, and Kuwait's first digital census reports were published from it. That is the outcome that matters. Not a platform that existed, but official statistics that came out of registers instead of out of the field.
I designed the architecture for the 2020 national programme on that basis, extending toward a wider set of government and private-sector data providers, and moved on before it ran. What the later programme eventually integrated, and how it ran, is the statistical authority's story rather than mine.
§ 07 — Rebuilding it in 2026
Everything above is what happened. This section is a redesign: what the same problem looks like now, and the architecture I would use for it today. It describes no work performed in Kuwait and no system in production.
The instructive part is how little of it changes.
The defects did not go away. They moved up the stack. The normalisation this project needed in 2010, folding alef variants, stripping tatweel, unifying Arabic-Indic digits, resolving ta marbuta, is documented as required preprocessing in current Arabic retrieval-augmented generation research, and it is what Arabic language models still run before anything else happens. What changed is the failure surface. In 2010, a name carrying an invisible character missed a database index. In 2026, the same name produces a different embedding vector, so the right document does not come back and nothing anywhere reports an error. The defect is identical and the fix is identical. It is now hidden inside a model instead of inside an index, which makes it harder to see rather than easier.
Occupational coding is the other place where that is true, and it is worth one sentence rather than a paragraph. Assigning a free-text job description to the right four-digit class is a task current classifiers still get wrong at rates that matter for official statistics, particularly at the leaf level where the distinctions are fine. The 2010 solution had no statistical component in it. It decomposed the description, routed each fragment to its level, and fell back to other registers as evidence, and it returned the same answer every time it ran on the same input. For a published national figure, reproducibility of that kind is not a nice property. It is the requirement.
The second half of the problem has not been solved either. Current work on linking Arabic names across heterogeneous databases describes that task as qualitatively harder than general Arabic named-entity disambiguation, for reasons the domain-general tools do not address: name forms of different depth, alternate naming conventions for the same person, and string similarity that collapses to near zero on pairs a human would match immediately. That was true across thirteen registers in 2011. It is true across an enterprise's systems today.
Where the field has arrived is worth stating carefully. The position now written down in the entity-resolution literature is that machine learning should improve speed and recall on ambiguous matches, and should augment rather than replace governance, explainability and expert review. Entity resolution stays a human-guided discipline supported by automation, rather than an unsupervised process nobody can inspect. That is the demographers' argument, restated by an industry that reached it independently fifteen years later.
The constraint that shaped all of it has not relaxed. It has spread. In 2010 the rule was that classified national records did not leave the ministry holding them. Today the same rule arrives as data residency, sovereignty and cross-border transfer requirements, and it lands on institutions that never thought of themselves as holding state data. The answer keeps the shape it had then. The tool is deployed inside the boundary and the data stays where it is, which in current terms means the model and the processing running within the tenancy, the region or the private estate the records are not permitted to leave, and an architecture where what crosses the boundary is a result rather than a corpus. What changed is that the temptation is stronger, because the convenient thing is now one API call away rather than one USB drive away.
What I would keep, which is most of it. The rules registry, and the discipline that every rule is visible and attributable to the person whose judgement it encodes. Lineage from a published figure back through every transformation to the source record. The reduction of many conflicting accounts to one adjudicated answer, with the adjudication itself recorded rather than implied. Reconciliation as its own stage rather than a step inside the load. Those were correct in 2010 because they were never really about technology.
What changes is the boundary. In 2010 the consumers of that register were statisticians and a reporting layer, and both were slow, human, and accountable. Today an AI system will read it, at machine speed, without the demographer's instinct for a number that looks wrong.
That moves three things from good practice to load-bearing controls.
The rules become policy-as-code rather than documentation. Thirteen hundred rules in a modelling tool are a description of the system. Thirteen hundred rules the pipeline executes, versioned, tested and diffable, are the system. The difference shows up the first time somebody asks why last quarter's figure changed.
Provenance has to survive retrieval. A statistical figure that cannot name the records it came from is an assertion. A model that quotes that figure without carrying the provenance with it has laundered an assertion into an answer. The lineage that existed for auditors now has to travel all the way to whatever consumes the number.
Access to the register becomes a governed tool boundary. An agent querying a national population register is exercising authority, and the governing questions are the ones I work on daily: which tools may be discovered, which may read and which may act, what the execution controls are, and what the audit record retains. A register is exactly the kind of asset where read-only, publicly declared, and never logging the caller's query are design decisions rather than defaults.
What I would not do is put a language model between the registers and the reconciliation. Adjudication has to be deterministic, replayable and attributable, because a census figure gets challenged years later and the institution has to reproduce it exactly. A model that resolves an entity differently on Tuesday than it did on Monday cannot support a published national statistic. Where a model earns its place is the layer above: helping a statistician find the cases worth looking at, drafting the explanation of a rule, surfacing the disagreements between registers that nobody has triaged. Judgement stays with the people who hold it, which is what the demographers were arguing for in 2011, and were right about.
§ 08 — Precedence
A system is worth revisiting fifteen years later only if you can say what was on the shelf at the time. Every date below is checkable against the issuing source. Two of the rows cut against the reading a sympathetic reader would reach for, and they are in the table for that reason, because a ledger that lists only the convenient dates is not a ledger.
| What the system did, 2010 to 2014 | When that capability was available to buy |
|---|---|
| Coded free-text job descriptions to a standard occupational classification | 2004. CASCOT, built by Jones and Elias at the University of Warwick's Institute for Employment Research. It codes to UK SOC and SIC by index matching with confidence scores, in English, to one classification at a time. |
| Held a classification estate as nodes and edges and walked it | 23 February 2010. Neo4j 1.0. Installing it meant new infrastructure inside a boundary that classified national records could not leave, and the processing had to go to the data. |
| Walked a property graph inside the database the records already sat in | Oracle's property graph support works with Oracle Database 12.2 and later. This ran on Oracle 11g, two major releases short. 11g carried RDF and OWL semantic graph support, which is triples for the semantic web rather than traversal of a four-level classification hierarchy. |
| Treated occupational and industrial coding as entities and relationships rather than as strings | May 2012. Google announced the Knowledge Graph, and "things, not strings" entered general enterprise vocabulary two years into this work. |
| Assigned codes by decomposition and traversal rather than by a trained model, returning the same answer on every run | Statistical and machine-learning occupation coding entered the official-statistics literature later. Gweon, Schonlau, Kaczmirek, Blohm and Steiner, "Three Methods for Occupation Coding Based on Statistical Learning", Journal of Official Statistics, 2017. |
| Resolved an Arabic description whose head noun and qualifier belong to different levels, into two four-level hierarchies at once, falling back to other registers as evidence where the string ran out | CASCOT, 2004, remains the nearest product. Arabic free text, two target hierarchies answering different questions, fragment-level routing, and cross-register fallback are each outside what it does. |
The dates are worth listing only because the constraint outlives them. Records that cannot leave the system holding them, classifications that have to reconcile across registers never built to agree, and a figure an institution has to defend years after publishing it: that combination did not end with this census, and it was never specific to censuses. It arrives now as data residency, as sovereign deployment, as a regulator asking how a number was produced. The tooling eventually reached the shape the constraint forces. The constraint was there first, and it will outlast this generation of tooling the way it outlasted the last.
What this case study does not claim
- No independently audited metrics. The data-quality figures, 12 percent at programme start and 99.5 percent at delivery, are from project records, not an external audit. Row volumes and entity counts are as documented at the time.
- No claim about the 2020 national programme's delivery. The system was architected for it. The programme's execution and eventual scope are the statistical authority's to describe.
- No claim of sole authorship. This was a nine-developer team inside a government programme with many contributors. What is described here is the architecture, the data integration design, the ISIC and ISCO mapping, the description standardisation, and the delivery direction.
- No AI, in any form. No models, no machine learning, no agents, and none of the categories in § 07 existed in this work. Python and statistical profiling for data repair is data engineering, and calling it anything else would be a retrofit.
- § 07 is a redesign, not a record. It describes no delivered system.