Why I did not want another ZIP: how the .ithz format emerged

When someone introduces a new archive format, the first question is usually simple: Is it smaller than ZIP?

The original question behind .ithz was different: Can an archive explain why it organized and compressed data in a particular way? And will it make the same decision if files are read in a different order or processed by more worker threads?

That question did not begin in archive design. It appeared earlier in ITHKOR kernel experiments. While studying finite information models, I encountered what initially looked like a small implementation detail: a local decision could depend on the order in which the system was traversed.

If an algorithm compresses or discards detail as soon as it sees a local signal, its decision can be fast. It may not be stable. A different legal traversal of the same system can encounter events in a different order and produce a different plan.

That was the seed of .ithz: not an attempt to rename existing compression, but an attempt to carry one research principle into real data handling.

Observe the structure first. Only then, at a clear checkpoint, decide what to compress and how.

The experiment that changed the direction

The pivotal result came from the C3b checkpoint stress test. In small graph models, it compared two selection strategies:

• budgeted visit responded continuously to local signals while visiting elements;
• layer checkpoint first collected information over an entire canonical layer and committed the decision only afterward.

In the broad contrast, the continuous budgeted_visit strategy achieved 0 out of 24 settings that passed every gate. layer_checkpoint passed 24 out of 24. A deeper checkpoint test then confirmed stability in 12 out of 12 monitored family-channel combinations.

These numbers do not establish a new law of physics. This was a finite diagnostic experiment. It did show something highly practical: when compression is meant to depend on structure rather than execution accident, a local peak is a poor place to make an immediate, binding cut.

The archive version of the problem is ordinary. Imagine a folder containing thousands of files. One program reads them by name, another in reverse order, and a third in parallel. If a compressor chooses the base for a group of similar files merely from whichever file it encounters first, the resulting plan may differ from one run to another.

The files can still restore correctly, yet the archive becomes harder to reproduce. Comparing builds becomes harder, auditing the decision becomes harder, and it is less obvious whether a difference came from changed data or merely a different work schedule.

Translating the idea into an archive

The kernel result became the ITHZ-CFC archive hypothesis: checkpointed compression guided by data structure.

Its process can be explained without equations:

1. Run a probe first. The archive observes data types, repetitions, similarity, entropy and estimated costs. It does not make a binding change yet.
2. Find supports and cuts. A support may be a family of similar JSON objects, logs, versions of one file, numeric columns or related binary regions. A cut says where it may be useful to separate a base, exact residual, metadata or literal escape stream.
3. Accumulate compression pressure. Estimated gain is compared with metadata and decoding cost. A local signal gets a vote; it does not issue an order.
4. Commit at a checkpoint. Only when a canonical layer closes is a stable plan selected.
5. Use reversible transforms only. If a file is represented as a base plus a residual, that residual must be exact. Decoded data must match the original bit for bit.

Techniques such as RLE, deflate, delta coding and byte shuffle are conventional. The unusual candidate is the layer above them: how data is grouped, where a cut is placed, when the decision becomes binding and how its stability is verified.

The first archive test

The first measured experiment used a small generated corpus: 19 files divided into 100 chunks. The same data was packed in original order, reverse order, hash order, two random orders and with different thread counts.

Across all seven variants, the checkpointed path produced the same compression plan, the same size and a correct lossless result. The greedy control also restored the data correctly, but five traversal orders produced five different plans.

That is a small but meaningful distinction. Both paths can pack and unpack data. Only one demonstrated, in this test, that its decision belonged to the data rather than to the order in which the data happened to arrive.

On selected similar layers, an exact base-and-residual representation saved 4,326 bytes against an estimate without the delta layer. At the same time, the complete first .ithz container was larger than the reference ZIP because it carried a large, inspectable manifest. This is essential to the story: the first success was not a compression-ratio victory. It was evidence for a stable plan, exact roundtrip and a measurable structural signal.

A later many-small-files specialization did beat ZIP/deflate on six of seven profiles in one test environment. tar.gz, however, remained smaller on all seven. .ithz therefore does not claim to be a universally superior compressor. Its narrower and more interesting statement is this: some data profiles benefit from deterministic structural planning, and the resulting decision remains inspectable.

What is unusual about .ithz

One hash for meaning, another for physical bytes

.ithz separates a semantic plan hash from the hash of the physical container. A later compatible format may store metadata more efficiently, changing the archive bytes. If the logical decision remains unchanged, the semantic hash can remain stable.

It is like giving a document new paper and binding while preserving its content and decision structure.

The archive can answer “why”

The native tool can show data families and explain why a particular file was assigned to a method: a small-text bundle, a similar version, incompressible content, or a guarded precompressed format.

This does not make the algorithm trivial. It means its decision does not have to remain hidden inside the final byte stream.

Safe verification does not need to write restored files

The verify safe mode decodes and checks headers, footers, the manifest, offsets, paths and SHA digests without first restoring the whole file tree. It blocks absolute paths, ../ traversal attempts, duplicates, reserved names and corrupted ranges.

Sometimes the correct compression is no compression

Photographs, videos, ZIP files and other already compressed or high-entropy data should remain in a guarded STORE mode when planning finds no safe gain. A forced “intelligent” transform might add only overhead and a marketing story.

How an archive became a memory zone

The archive format later met a second problem: AI-agent memory.

An agent needs more than a pile of notes. It needs to know which decision is current, which claim is blocked, which test is a gate, what changed, and what must not silently disappear when context is shortened.

This is where the archive idea joined working memory. A project.ithz zone can preserve checkpoints, decisions, risks, allowed and blocked claims, reviewer notes and derived indexes. This is no longer only file compression. It is compression of a project's story without losing its decisive boundaries.

The same principle repeats:

• do not store everything without distinction;
• do not decide only from the latest event;
• preserve an exact trail of what was selected;
• let the next person or agent verify the result.

What could follow from this

If the direction continues to hold, .ithz may become useful wherever restoring files alone is not enough.

Scientific data may need to reproduce not just an output but also the plan that partitioned it. Software builds may use a stable semantic hash to distinguish a real change from accidental scheduling differences. Audits may use “why” explanations to show why content was compressed, preserved or blocked. AI-agent memory may remain local, versionable and readable without trusting an invisible cloud history.

The broadest consequence may be conceptual: compression is not only about how many bytes remain. It is also about which distinctions a system considers important, and whether that decision can later be inspected.

Not a magic format, but an honest experiment

.ithz is not a replacement for every archive today. It does not beat ZIP, tar.gz, 7z or zstd on every data type. Some experimental transforms produced no gain and were not promoted into the default format. That is part of the intended development process, not a flaw in the story.

What makes .ithz unusual today is elsewhere: it joins lossless compression with a canonical plan, safe verification, explainable decisions and inspectable memory. It is an archive that does not merely close data inside a container. It also tries to preserve the answer to a question:

Why was this data organized in this particular way — and can we reproduce the same decision?

That is why the format emerged. Not because the world needed one more file extension, but because compression and memory could be made a little less dependent on accident and a little more open to inspection.

Continue with ITHZ Drive, the technical ITHZ Native layer, or the experiment map.