Consider a team maintaining a Django monolith of 80,000 lines directed to migrate to a set of FastAPI microservices — one per business domain. The migration involves identifying domain boundaries, extracting modules, replacing in-process function calls with HTTP or message-queue calls, generating Dockerfiles, and updating routing configuration. Each of these steps is performed manually, inconsistently, and unreproducibly. There is no artifact that describes what the target codebase should look like structurally, and no way to verify that an intermediate state is valid.
The same pattern appears across many common tasks: renaming all exported symbols to follow a new convention, splitting a flat C project into a header/source pair per module, converting CommonJS to ES Modules, generating TypeScript type declarations from a SQL schema. Each is a structural transformation of a codebase — it changes how code is organized, not what it computes.
Tools like jscodeshift, ast-grep, and sed-based scripts address
fragments of this problem through local rewrites. They do not address the global structural question —
what shape should the entire codebase be in — nor do they let an engineer specify, compose, or
verify whole-codebase structural migrations. The absence of a formal model for codebase shape has a direct
practical consequence: migrations are underdetermined. There is no specification to check against, no way to
detect that a step has left the codebase in a structurally invalid intermediate state, and no basis for
automating or learning the transformation.
CodeFlock addresses this by treating codebase structure as a typed object. Once a codebase has a formal shape — a Canon — it becomes possible to define what migrations are valid, to compose them from reusable building blocks, and critically, to train a model to recognize shapes and plan migrations between them. The building block strategy is not just an engineering convenience: it produces a structured vocabulary of codebase shapes and the transitions between them that can drive both static verification and machine learning.
A Canon is not a type system for the code inside files — it is a type system for the arrangement of files, modules, and components. Two projects can contain identical logic but conform to different Canons. Crucially, Canon conformance is checkable: given a project and a Canon, a validator can decide P ⊨ C without executing the project.
apps/ directory. All business logic
lives in models.py, views.py, serializers.py per app. Inter-app
communication is direct Python import. No Dockerfile per module.main.py, routes/, schemas/, a Dockerfile, and
communicates with other services only via HTTP or a message queue — no cross-service Python import.
A Canon is a predicate on a project, not a recipe for building one. The same predicate can be checked against any project to determine conformance, making it reusable across migrations.
Retcons compose: if R₁ : A → B and R₂ : B → C, then R₂ ∘ R₁ : A → C is a valid Retcon. Complex migrations are built from sequences of simpler ones.
B₁ cluster_by_call_graph : Monolith → Clustered
(identify domain boundaries via call-graph analysis)
B₂ extract_clusters : Clustered → Extracted
(move each cluster to its own directory)
B₃ replace_cross_imports : Extracted → Decoupled
(replace in-process calls with HTTP stub calls)
B₄ add_service_entrypoint : Decoupled → ServiceReady
(add main.py + FastAPI router per service)
B₅ generate_dockerfile : ServiceReady → Microservice
(emit Dockerfile + requirements.txt per service)
Retcon: MonolithToMicroservice = B₅ ∘ B₄ ∘ B₃ ∘ B₂ ∘ B₁
Each intermediate Canon — Clustered, Extracted, Decoupled, ServiceReady — is a formal predicate that can be
checked after each step.
The choice of plane is an algorithmic decision. The operation cluster by domain boundary cannot be performed efficiently on the File plane — it requires the Graph plane (a call graph of functions across files). The operation rename all exported symbols cannot be performed correctly on the Text plane without false positives — it requires the AST plane.
| Plane | Representation | Operations enabled | Operations impractical |
|---|---|---|---|
| File | Paths and byte content | Move, copy, delete, create files; repackage directory structure | Rename a symbol; find callers; detect unused imports |
| Text | Sequence of characters | Regex search/replace; delimiter-split; string interpolation | Structure-aware rename; cross-file reference tracking |
| Token | Lexical token stream | Token-type filtering; count identifiers; detect keywords | Structural queries (scope, nesting, call targets) |
| AST | Abstract syntax tree | Rename with scope awareness; extract function; inline variable; detect unused exports | Cross-file call graph; dynamic dispatch resolution |
| Graph | Call graph (nodes = functions, edges = calls) | Find strongly connected components; topological sort; cluster by reachability; detect cycles | Inspect syntax of individual functions |
| Service | Deployable units with endpoints | Generate Dockerfiles; define API contracts; configure routing | Inspect source code structure |
The decomposition captures a practical engineering principle: the difficulty of a transformation is determined by the plane, not the Canon. Moving to a richer plane makes hard operations easy; moving back makes the result portable.
cluster_by_call_graph) is a Vertical Retcon:A Universe bounds the vocabulary of a migration. A team working on Python projects defines a Python Universe; a team working on frontend repositories defines a Frontend Universe. Universes can share Building Blocks where operations are language-agnostic.
This example crosses language boundaries entirely — source is SQL DDL, target is TypeScript — making the role of Canons and Planes concrete.
Source Canon (SQL-DDL): One .sql file. Contains only CREATE TABLE
statements. Each column has a SQL type. Foreign keys declared with REFERENCES.
Target Canon (TS-Types): One .ts file. One exported TypeScript
interface per table. Each column maps to a TypeScript property. Foreign key columns reference
the corresponding interface type. No any. File compiles without errors.
The Retcon SQLToTypeScript is a Vertical Retcon:
X_down: Parse SQL DDL text → schema graph
(Text plane → Graph plane)
Nodes = tables, edges = foreign-key relations
Canon unchanged: still SQL-DDL shape, now in graph form
R_lateral: Walk schema graph in topological order
(Graph → Graph, now conforming to TS-Types-Graph Canon)
Map SQL types: VARCHAR → string, INT → number,
BOOLEAN → boolean, TIMESTAMP → Date, nullable → T | null
Replace FK edges with interface reference edges
X_up: Serialize graph nodes as TypeScript interface declarations
(Graph → File plane, TS-Types Canon)
Emit one interface per node, properties in column order
The intermediate Graph plane is essential: topological sort on the schema graph ensures that no interface references a type not yet declared. This ordering is impossible to determine correctly on the Text plane without re-implementing a parser.
A composed Retcon can be verified before any code is transformed. Two properties are checked:
Canon-chain validity: For R = Bₙ ∘ … ∘ B₁, the output Canon of Bᵢ must equal the input Canon of Bᵢ₊₁ for all i.
Plane-continuity: When the output Plane of Bᵢ differs from the input Plane of Bᵢ₊₁, a Crossover must appear between them. A composition that switches planes without a Crossover is ill-formed.
These two checks give CodeFlock a type system over transformations — one that surfaces errors before the system touches any file on disk. This matters most in long migrations: a malformed step at position 7 of a 12-step chain otherwise produces a codebase that is in neither the source nor the target Canon, with no record of where it diverged.
| Error type | Example | Detection phase |
|---|---|---|
| Canon mismatch | B₂ expects Clustered Canon but B₁ outputs Monolith Canon | Static (before execution) |
| Missing Crossover | B₁ outputs Graph plane, B₂ expects File plane, no Crossover between them | Static (before execution) |
| Canon violation | B₃ output fails validate() for Decoupled Canon | Runtime (after B₃ executes) |
| Crossover loss | X_down is lossy; X_up cannot reconstruct needed information | Runtime or design-time review |
The CodeFlock model was designed as an engineering framework, but its structure has an underexplored consequence: it generates well-formed training data for machine learning, and in doing so opens several distinct learning problems. This section formalizes what can be learned from a corpus of Canons and building block traces.
Every execution of a building block Bᵢ : Canon_A → Canon_B on a real project produces a triple:
(P_in, label(Bᵢ), P_out) where P_in ⊨ Canon_A and P_out ⊨ Canon_B
This is a supervised training example by construction. The input is a project in a known structural state. The output is the same project in a different structural state. The label is discrete and unambiguous — the name of the building block applied. Collected across many real codebases, this produces a dataset of (before-state, transformation-type, after-state) triples at a granularity that no existing code dataset provides. Current code datasets contain files and commits; they do not label what structural transformation a commit performs.
A migration chain B₁, B₂, …, Bₙ produces not one training signal but n intermediate project states, each conforming to a known Canon. This structure maps directly onto curriculum learning: rather than training a model on the full migration in one shot, it can be trained on each step in sequence, with Canon conformance serving as the reward signal at each checkpoint.
The intermediate Canons also define a natural difficulty ordering on the training data. Early steps (e.g. annotating domain boundaries) are local and recoverable. Late steps (e.g. generating Dockerfiles) depend on all prior steps being correct. A curriculum that sequences training examples by step index mirrors the dependency structure of the transformation.
Canon conformance — the predicate P ⊨ C — is a binary classifier with a formal definition. A model trained to predict conformance can serve as a learned verifier, replacing the hand-written rule-checker for Canons that are too complex or context-dependent to specify exhaustively. More importantly, a model that predicts conformance has implicitly learned an embedding of codebase structure: two projects that conform to the same Canon should be nearby in the learned representation space, regardless of language, size, or domain.
The same project represented in multiple planes — File, AST, Graph — constitutes a multimodal view of the same object. This is directly analogous to multimodal learning in vision and language: the same content expressed in different modalities, with cross-modal alignment as a training objective. In the CodeFlock setting, a Crossover from File to AST is a deterministic translation between modalities. A model trained on both the File and AST representations of the same project, with the Crossover as alignment, learns to ground structural descriptions at two levels of abstraction simultaneously.
This representation also supports cross-plane retrieval: given a project in the Graph plane (a call graph), retrieve the most structurally similar project in the File plane (an actual directory layout). This is a non-trivial retrieval problem — it requires the model to understand that structural similarity in one plane implies structural similarity in another.
A Universe is a closed set of Canons and building blocks. A model generating a migration plan is constrained to output sequences of building blocks that (a) belong to the Universe and (b) form a valid Canon chain. This is a constrained generation problem — structurally identical to grammar-constrained decoding in language models, where the output must conform to a formal grammar at each decoding step.
The Canon graph of a Universe — where nodes are Canons and edges are building blocks — defines the valid transition system. A migration planner is a model that performs reachability search over this graph, guided by a learned heuristic. The heuristic can be learned from observed migration traces: which paths between Canons are taken most often, which building blocks tend to appear together, which intermediate Canons are most commonly used as waypoints.
| Learning problem | Input | Output | ML formulation |
|---|---|---|---|
| Block classification | (P_before, P_after) pair | Building block label | Multi-class classification |
| Canon recognition | Project P in any plane | Canon C ∈ U | Multi-class classification with structured label space |
| Next block prediction | Project P, target Canon C_t | Next building block Bᵢ | Autoregressive sequence prediction |
| Migration planning | C_source, C_target, Universe U | Block sequence B₁…Bₙ | Constrained structured prediction / graph search |
| Cross-plane alignment | Project in Plane A | Embedding shared with Plane B | Contrastive learning |
| Canon conformance | Project P, Canon C | P ⊨ C (binary) | Binary classification / learned verifier |
These six problems are not independent. A model that solves Canon recognition has learned a representation useful for block classification. A model that solves cross-plane alignment has learned embeddings that support migration planning. The CodeFlock formalism ties them together: they are all facets of the same underlying structure, and progress on one informs the others.
This document was prepared as a technical overview to support a discussion with an academic supervisor. The definitions are presented informally but with enough precision to anchor a more rigorous formalization. The examples use Python/Django and SQL/TypeScript as concrete instances; the model is language-agnostic.