ZOSCII MQ System Documentation

1. Overview and Advantage

The World's First Truly Quantum Proof Message Queue (MQ) and Data Store when used in conjunction with ZOSCII.

ZOSCII MQ is a minimal, file-based Message Queue (MQ) designed for high-assurance, B2B asynchronous communication. It leverages the file system for durability and uses specific file naming conventions for chronological ordering and message retention. It also can provide a robust message store service.

Why Use ZOSCII MQ?

The architecture offers key advantages over traditional message brokers (like RabbitMQ or Kafka) in specific freight and logistics scenarios:

• High Durability (Disk-First): Messages are written directly to the file system. Durability is tied directly to the operating system and disk health, ensuring messages are safe immediately upon publishing.
• Asynchronous B2B Integration: It excels at decoupling systems. The Puller (replikate.php) model allows external partners to control their consumption rate and timing, minimizing your exposure to their network failures.
• Security Integration (ITS): The file-based nature is perfectly suited for integrating Information Theoretic Security (ITS) encoding, where the message payload is encoded before being written to disk, offering a higher level of data security than standard SSL/TLS alone. The server needs no knownedge of the message content.
• Simplicity and Control: There is no complex daemon, database, or broker to manage. All state is controlled via files, making it easy to audit and debug.
• Zero Knowledge: When used with an ITS encoding such as ZOSCII, there is no information available at the server. No meta data besides the queue name, date/time of storage and retention.

2. Usage for Integrators: Reading and Writing Messages

ZOSCII MQ uses a simple HTTP API (via index.php) for publishing and fetching messages, and a cron script (replikate.php) for B2B integration.

A. Publishing Messages (Writing to a Queue)

Messages are published using an HTTP GET request to the index.php endpoint.

Endpoint:

/index.php?action=publish

JSON Responses:

The main ZOSCII MQ index.php gives back it's responses as either binary data with a filename as it's Content-Disposition *, or as JSON messages in the format shown here. If possible it is always good to check the system. The error if it has a value is important. If there is no value in the error, then the result is valid. A message may be provided regardless of errors or results.

{
	"system":"ZOSCII MQ",
	"version":"1.0",
	"error":"",
	"message":"Message published.",
	"result":[]
}

* Refer to the implementation of the fetchNextMessage function within replikate.php for an example of how to handle the binary and JSON responses.

* Note: It is possible to confuse JSON messages stored in ZOSCII MQ vs those JSON messages from ZOSCII MQ itself. However it shouldn't hapen if you first check the filename from within the Content-Disposition. Checking if response.system is present and whether it is "ZOSCII MQ" will make this more robust.

Example Command (HTTP):

GET /index.php?action=publish&q=customer_b&r=30&msg=%7B%22sku%22%3A%20%22123%22%7D&n=a1b2c3d4-e5f6-4000-8000-0123456789ab

JSON Response:

{
	"system":"ZOSCII MQ",
	"version":"1.0",
	"error":"",
	"message":"Message published.",
	"result":[]
}

Filename Format:

The message is saved as a file in the queues/QUEUE_NAME/ directory with the format: YYYYMMDDHHNNSSCCCC-RRRR-GUID.bin

B. Fetching Messages (Reading from a Queue - API Pull)

Messages are fetched sequentially using the unique filename as a pointer. The consumer must track this pointer (the after parameter) to ensure guaranteed, chronological delivery.

Endpoint:

/index.php?action=fetch

Success Response:

• The consumer must store the returned filename (Content-Disposition header) as its pointer.
• The consumer must only advance the pointer after successfully processing and acting on the message content.
• The system performs no message acknowledgment (ACK). Delivery guarantee relies entirely on the consumer's proper use of the pointer (after parameter).

C. B2B Replication (Puller)

For external partners (e.g. ABC Ltd) to pull data from your (e.g. XYZ Ltd) queue, they should run a dedicated cron job on their server (e.g. ABC_LOCAL_SERVER).

Script:

replikate.php

Setup:

• The partner must install this script on their server.
• They should set up a scheduled task (cron job or Windows Scheduler) to run the script every few minutes.

Endpoint:

/replikate.php

Example Command for Queue to Queue replication (HTTP):

GET /replikate.php?url=https://SOURCESERVER/index.php&sq=SOURCEQUEUE&tq=TARGETQUEUE

The script manages the state/pointer file (replikate_state_SOURCEQUEUE.txt) automatically.

Example Command for Queue to Store replication (HTTP):

GET /replikate.php?url=https://SOURCESERVER/index.php&sq=SOURCEQUEUE

D. Storing Messages (Writing to a Data Store)

Messages are stored using an HTTP GET request to the index.php endpoint.

Endpoint:

/index.php?action=store

Example Command (HTTP):

GET /index.php?action=store&r=30&msg=%7B%22sku%22%3A%20%22123%22%7D&n=a1b2c3d4-e5f6-4000-8000-0123456789ab

JSON Response:

{
	"system":"ZOSCII MQ",
	"version":"1.0",
	"error":"",
	"message":"Message stored.",
	"result":"202511021826110000-0030-9cc1980a-1719-440a-9650-6991c9cba11d.bin"
}

Filename Format:

The message is saved as a file in the store/x/y/z/ directory with the format: YYYYMMDDHHNNSSCCCC-RRRR-GUID.bin. The file will be returned if the storing of the message is successful. You will need to note this filename for later fast retrieval.

For later retrieval of the message, you must note the response.result.

E. Retrieving Messages (Reading from a Data Store)

Messages are retrieved from the data store by using the previously returned filename. The consumer must track this filename if fast retrieval is required.

Endpoint:

/index.php?action=retrieve

Success Response:

• The consumer must read the returned filename (Content-Disposition header) to determin if the file was fetched.
• If the filename was not fetched, a JSON result may have been returned.

JSON Error Response:

{
	"system":"ZOSCII MQ",
	"version":"1.0",
	"error":"Invalid name 'x202511021826110000-0030-9cc1980a-1719-440a-9650-6991c9cba11d.bin'.",
	"message":"",
	"result":[]
}

F. Scanning Messages (Scan the Data Store for unidentified messages)

Messages are retrieved from the data store by using the previously returned filename. The consumer must track this filename if fast retrieval is required.

Endpoint:

/index.php?action=scan

Success Response:

{
	"system":"ZOSCII MQ",
	"version":"1.0",
	"error":"",
	"message":"",
	"result":[
		"202511021259550001-0005-5192e106-7a99-4418-98eb-8d4cb4c6aa3a-u.bin",
		"202511021259550001-0005-be2b00c1-9419-4661-8c1d-2219780c7980-u.bin"
	]
}

• The -u within the message name indicates the message has not yet been identified.
• Given the response array of unidentified files, you can now fetch them, identify them and flag them as identified.

G. Identify Messages

Once messages have been scanned for, then identified, they can then be flagged as identified.

Endpoint:

/index.php?action=identify

Example Command (HTTP):

GET /index.php?action=identify&names=["202511021259550001-0005-5192e106-7a99-4418-98eb-8d4cb4c6aa3a-u.bin","x202511021259550001-0005-be2b00c1-9419-4661-8c1d-2219780c7980-u.bin"]

Success Response:

{
	"system":"ZOSCII MQ",
	"version":"1.0",
	"error":"",
	"message":"Returned messages identified.",
	"result":[
		"202511021259550001-0005-5192e106-7a99-4418-98eb-8d4cb4c6aa3a.bin"
	]
}

• The messages successfully identified will be returned with the -u removed.

3. Server Setup and Maintenance (For Administrators)

A. Core Directory Structure

The system is file-based and requires a dedicated root directory:

./
|-- index.php             (HTTP API: Publish/Fetch)
|-- replikate.php         (Cron: External Puller)
|-- claireup.php          (Cron: Retention Cleanup)
|-- nonce/                (Optional nonce files go here)
|-- queues/
|   |-- customer_a/       (Queue 1)
|   |   |-- *.bin
|   |-- customer_b/       (Queue 2)
|   |   |-- *.bin
|-- states/               (State Files are here for Replicators)
|   |   |-- replikate_state_*.txt
|-- store/                (Data Store Files are here)
|-- statissa.php          (Statistics, it's better to place statissa elsewhere)

B. Critical Cron Jobs

Two cron jobs must be set up on the ZOSCII MQ server to ensure stability:

1. Message Retention Cleanup (claireup.php)

This job ensures disk space is managed by deleting expired messages based on the **RRRR** value in the filename. It processes **all queues globally**.

Cron Example:

Although it's possible to run claireup.php from a browser it is recommended to run it from cron if possible. To run it from the browser, you will first need to change the CLI_ONLY constant to FALSE from within the claireup.php file. If you do that, you are best ensure that it cannot be run by unauthorised parties if you don't want it to run.

0 0 * * * /usr/bin/php /path/to/claireup.php

4. Deployment Strategies

ZOSCII MQ supports two primary deployment patterns, each optimized for different use cases:

Strategy 1: Internal Business Message Bus

Primary Server (HQ)          Backup Server (DR Site)
   ZOSCII MQ        ─────→      ZOSCII MQ
   [All Queues]     replikate   [Mirror Copy]
                    (periodic)
    

Message Distribution Patterns

              Primary Server
                 ZOSCII MQ
          [company_announcements]
                    ↓
       ┌────────────┼────────────┬────────────┐
       ↓            ↓            ↓            ↓
    John's PC    Mary's PC    Bob's PC    ... 1,000+ PCs
    
1 message published → 1,000+ PCs all fetch from same queue
Each PC independently reads the same messages
No replication needed - standard ZOSCII MQ behavior
    

                Primary Server (HQ)
                   ZOSCII MQ
            [company_announcements]
                      ↓
          ┌───────────┴───────────┬───────────────┐
          ↓                       ↓               ↓
     replikate.php          replikate.php    replikate.php
          ↓                       ↓               ↓
    Server A (Sydney)       Server B (Tokyo)  Server C (London)
      ZOSCII MQ               ZOSCII MQ         ZOSCII MQ
          ↓                       ↓               ↓
    ┌─────┼─────┐           ┌─────┼─────┐   ┌─────┼─────┐
    ↓     ↓     ↓           ↓     ↓     ↓   ↓     ↓     ↓
  PC1   PC2  PC500       PC501 PC502 PC1000 ... PC2000 PC3000

1 message → striped to 3 regional servers → 3,000+ PCs globally
Each regional server services local PCs (lower latency)
    

Strategy 2: High Throughput AWS B2B Integration

Business A (AWS)         Business B (AWS)         Business C (AWS)
 ZOSCII MQ Docker         ZOSCII MQ Docker         ZOSCII MQ Docker
 [Push anytime]           [Push anytime]           [Push anytime]
      ↓                         ↓                         ↓
      └─────────────────────────┴─────────────────────────┘
                                 ↓
                    Head Office Server (Local)
                         ZOSCII MQ
                    [Pulls 24/7 via replikate.php]
                    [Aggregates all business data]
    

Strategy Comparison

5. Docker Deploys

Deploying ZOSCII MQ using Docker provides an **instant, persistent message queue** solution. There are three primary deployment strategies for managing the deployment environment and persistent queue data (messages and nonces):

Strategy 1: Instant Deploy (Docker Named Volumes) - Recommended for simplicity

This strategy is best for quick setup and development environments. It uses **Docker Named Volumes** which Docker automatically manages on the host machine.

Deployment Details:

• Persistence: The default docker-compose.yml uses Named Volumes (zoscii_data and zoscii_nonce).
• Location: Data is stored in a hidden, internal location managed by Docker on your host machine.
• Expansion: The storage size is dynamic and will grow automatically, limited only by the host machine's available disk space.

The volumes section in the docker-compose.yml looks like this:

volumes:
  # Data is mapped to Docker-managed volumes defined at the bottom of the file.
  - zoscii_data:/app/queues
  - zoscii_nonce:/app/nonce

volumes:
  zoscii_data:
  zoscii_nonce:

Strategy 2: Dedicated Data Path (Host Bind Mounts) - Recommended for local production

This strategy gives you full control over where the queue data resides. This is essential for environments where you need to explicitly back up, monitor, or manage the queue files using host-level tools (like rsync or specific backup agents).

Deployment Details:

• Persistence: You explicitly map a known local directory on the host to the container's data path.
• Location: Data is stored in the specific folder you choose (e.g., ./zoscii-data/).
• Control: Provides the ability to "point" the queue storage to a dedicated folder, external drive, or network file system.

The volumes section in the docker-compose.yml is modified to specify the host path:

volumes:
  # The host path (./zoscii-data) is explicitly mapped to the container path.
  - ./zoscii-data/queues:/app/queues
  - ./zoscii-data/nonce:/app/nonce

# NOTE: Named volumes are NOT defined when using Bind Mounts.

Strategy 3: Serverless Cloud Deploy (AWS Fargate/Google Cloud Run) - Recommended for zero local install

If you wish to deploy without installing Docker or running a package manager locally, the best method is a **Serverless Container Service** that manages the container's lifecycle and infrastructure.

Deployment Details:

• Zero Install: You only interact with the cloud provider's web console (AWS/Google Cloud).
• Managed Service: The cloud provider runs your container, manages the host OS, and handles auto-scaling.
• Storage Configuration: For persistence, the container's /app/queues directory must be mounted to a cloud file system service (like **AWS EFS** or **Google Cloud Filestore**) during deployment setup. This replaces the Docker volume concept with a network file system.

Once your ZOSCII MQ Docker image is pushed to a public registry (via automated build, e.g., GitHub Actions), you can deploy it in a few clicks.

6. Running the Cleanup Cron Job (claireup.php)

Since claireup.php is designed to be run from the command line (CLI), you can execute it manually inside the running container:

docker-compose exec zoscii-mq php /app/claireup.php

You can automate this by setting up a host-level cron job that calls the docker-compose exec command every minute. **Note for Strategy 3 (Cloud Deploy):** For cloud deployments, you must use the cloud provider's native scheduling tools (e.g., **AWS CloudWatch Events/EventBridge** or **Google Cloud Scheduler**) to execute the cleanup command on the running container.

6. Comparison with Traditional Message Brokers

Architecture Comparison

Performance and Scale

Security Comparison

Use Case Suitability

Federated Architecture Advantage

ZOSCII MQ's replikate.php enables a unique federated architecture:

Company A (Sydney)     Company B (Tokyo)      Company C (London)
    ZOSCII MQ    ←──→    ZOSCII MQ    ←──→    ZOSCII MQ
         ↓                    ↓                     ↓
   [Own Control]        [Own Control]         [Own Control]
   [Own Security]       [Own Security]        [Own Security]
   [Own Data]           [Own Data]            [Own Data]

Each node operates independently - no central point of failure, no shared infrastructure, perfect for B2B logistics and freight forwarding where different organizations need to maintain sovereignty over their systems.

Total Cost of Ownership (TCO)