IoT Messaging Framework Architecture

For Physical Access Control & Building Automation

1. Architecture Overview

This document outlines the architecture for an enterprise IoT platform focused on physical access control and building automation. The platform is built on open standards, uses a robust messaging architecture, and leverages modern infrastructure technologies to provide a secure, scalable, and maintainable solution.

1.1 Design Principles

Standards-based: Uses well-defined message structure and topic patterns
Resilient: Designed to operate during network outages
Secure: Multi-layered security approach
Scalable: From single-site to multi-site deployments
Edge-first: Processing at the edge for low latency
Open: Based on open standards and technologies

1.2 Key Components

NATS: Core messaging system with JetStream persistence
Nebula: Overlay network for secure communication
FreeBSD: Base operating system with ZFS
TimescaleDB: Time-series data storage
Caddy: Web server and reverse proxy
Edge Nodes: Distributed processing units

2. Core Infrastructure

2.1 Network Architecture

                                     +----------------+
                                     |                |
                                     |  rsync.net     |
                                     |  (Backups)     |
                                     |                |
                                     +-------^--------+
                                             |
                                             | ZFS Snapshots
                                             |
+----------------+     HTTPS     +----------v---------+
|                |-------------->|                    |
|  End Users     |               |  Caddy Reverse     |
|  Web Browsers  |<--------------|  Proxy            |
|                |               |                    |
+----------------+               +----------^---------+
                                            |
                                            | Local Network
                                            |
+----------------+               +----------v---------+     +----------------+
|                |  Nebula      |                    |     |                |
|  Edge Node     |<------------>|  Central Server    |     |  Zabbix        |
|  NATS Leaf     |  Overlay     |  NATS Cluster      |<--->|  Monitoring    |
|                |  Network     |  TimescaleDB       |     |                |
+----------------+               +--------------------+     +----------------+
       ^
       |
       | Local Connection
       |
+------v---------+
|                |
|  IoT Devices   |
|  "Things"      |
|                |
+----------------+

2.2 Messaging Foundation

NATS serves as the core messaging system
Standardized topic structure: {org_id}.{edge_code}.{thing_type}.{thing_code}.{message_type}
JetStream for message persistence and guaranteed delivery
Leaf nodes for edge connectivity with independent operation capability

2.3 Data Storage

TimescaleDB for time-series data storage
Local ZFS filesystem for data integrity and snapshots
Backup strategy using ZFS snapshots to rsync.net

3. Security Architecture

3.1 Network Security

Nebula overlay network providing encrypted communication between nodes
Nebula certificates for node authentication and network-level access control
Host-based firewalls restricting access to only required services

3.2 Service Authentication

NATS Authentication:
User/password with bcrypt for local device connections
NKeys for leaf node connections to main cluster
Topic-level permissions enforced via NATS authorization
Web Layer:
Caddy with proper TLS certificates for UI and API gateway
OAuth2 or similar for user authentication

3.3 Local Security

TimescaleDB and NATS bridge services communicate locally on the same host
Localhost-only binding for services not needing external access
Principle of least privilege for service accounts

3.4 Edge Security

Edge installations run NATS leaf nodes connecting over Nebula network
Devices connect to edge nodes using NATS user/password authentication
Fallback HTTP-to-NATS transformer behind Caddy for devices that can't use Nebula

4. Component Workflow

4.1 UI and API Layer

Web UI and API gateway behind Caddy with proper TLS certificates
API endpoints for configuration and management
User authentication and authorization integrated with enterprise systems

4.2 Core Services

NATS Cluster:
Message routing and delivery
JetStream for persistence
Subject-based access control
TimescaleDB:
Historical time-series data storage
Optimized for IoT event volumes
Hypertable partitioning and retention policies

4.3 Edge Services

NATS Leaf Nodes:
Connect to main cluster using NKeys
Operate independently during connectivity loss
Synchronize messages via JetStream on reconnection
Edge Processing:
Local rules engine for immediate actions
Credential caching for offline operation
Local event buffering during outages

4.4 Device Communication

Devices connect to edge nodes over Nebula network
Authentication via NATS user/password
Topic permissions enforced at connection level
Fallback HTTP proxy with TLS certificates for devices without Nebula

5. Data Flow

5.1 Device Events

Device generates an event (e.g., sensor reading, access attempt)
Event is published to NATS on the edge node with standard message format
Edge node processes event with local rules for immediate actions
Event is persisted in local JetStream
Event is forwarded to central NATS cluster (when connected)
Event is persisted in TimescaleDB for historical record

5.2 Commands and Configuration

Command or configuration change initiated through UI/API
Change is validated and stored in central configuration
Command is published to appropriate NATS subject
Edge node receives command via NATS leaf node connection
Edge node forwards command to target device
Acknowledgment flows back through the same path

6. Operational Aspects

6.1 Monitoring and Alerting

Zabbix for infrastructure and service monitoring
Structured logging across all components
Alerting for service disruptions and security events
Performance metrics collection and visualization

6.2 Maintenance and Updates

Ansible for automated certificate management:
Caddy TLS certificate renewal
Nebula certificate issuance and rotation
Infrastructure updates:
FreeBSD package updates
Service deployment and configuration
Database schema migrations

6.3 Backup Strategy

ZFS snapshots for point-in-time recovery
Scheduled syncing of snapshots to rsync.net
Regular testing of restoration procedures

7. Scalability Considerations

7.1 Horizontal Scaling

NATS cluster can scale with additional server nodes
TimescaleDB can be scaled with read replicas or partitioning
Edge nodes can be added to support new locations

7.2 Performance Optimization

Edge processing reduces central server load
JetStream provides efficient message delivery
TimescaleDB hypertables optimize time-series queries
Caching strategies for frequently accessed data

8. Future Enhancements

8.1 Security Enhancements

Application-level mutual TLS (mTLS) for service-to-service communication
Enhanced authentication mechanisms
Automated security scanning and vulnerability assessment

8.2 Operational Improvements

Observability pipeline with structured logging
Enhanced disaster recovery testing
Configuration versioning in Git
Automated edge device provisioning

8.3 Feature Additions

Circuit breakers for external dependencies
Feature flag system for gradual rollout
Performance baseline establishment
Health check endpoints for all services

9. Conclusion

This architecture provides a solid foundation for an IoT messaging framework focused on physical access control and building automation. It balances security, reliability, and performance while maintaining operational simplicity suitable for an MVP deployment. The design allows for future enhancements while meeting current requirements with a pragmatic approach to security and infrastructure management.