Application evidence report
Chat-room application case study
Abstract
For one moderated two-room workflow, observable behavior matched across Real Socket.IO 4.8.3, published Smocket 0.4.2, and a handwritten mock, while the owned test-support surface differed. This report compares that surface and the pinned implementation evidence behind the result.
- Recorded
- 2026-08-12
- Environment
- darwin arm64 · Node v22.16.0
- Targets
- Socket.IO 4.8.3 · Smocket 0.4.2 · handwritten
These observations apply only to the recorded chat-room workflow and are not a claim of overall Socket.IO compatibility.
01
Research question & method
What does each approach own to support the same selected application test?
All three targets run the same application, scenario, and assertions. Only dependency wiring and bootstrap change; the handwritten fixture additionally owns the mock being compared. No shared file contains a target branch or workaround.
Listeners register before their actions. Acknowledgements and later per-socket markers—not delays or timeouts—establish completion and non-receipt. The same assertions execute twice per target in one process.
| File | Responsibility | Observed surface | Evidence |
|---|---|---|---|
app.js | Chat-room application handlers | 86 physical source lines | Pinned source ↗362afdbb7d9c… |
scenario.js | Shared workflow orchestration | 183 physical source lines | Pinned source ↗1ebe7269a18b… |
assertions.js | Expected observation and deep equality | 98 physical source lines | Pinned source ↗1636bb1a4995… |
02
Authored support surface
Exact target-owned JavaScript measured for this recorded workflow.
| Question | Real Socket.IO | Exact published Smocket | Handwritten mock |
|---|---|---|---|
| Exact dependencies / clean install | socket.io@4.8.3, socket.io-client@4.8.3 | smocket@0.4.2 | None |
| Bootstrap / runtime setup | Local HTTP server, real Socket.IO server/client, and an ephemeral loopback TCP port. | Published smocket package with an in-memory Server/connect bootstrap. | No package dependency; an in-memory fixture imports its own Socket.IO-shaped mock. |
| HTTP server / port ownership | Fixture owns HTTP server creation, listen(0), address lookup, and the port. | No HTTP server or port setup in the fixture. | No HTTP server or port setup in the fixture. |
| Client activation / shutdown | Fixture disables auto-connect/reconnection, activates each client, and closes the server. | Connection is available in memory; activate is empty and the fixture closes Smocket. | Connection is available in memory; activate is empty and the fixture closes its server. |
| Authored fixture files | bootstrap.js (61) | bootstrap.js (28) | bootstrap.js (28) + handwritten-socket-io.js (212) |
| Application-owned mock | No application-owned mock implementation. | No application-owned mock implementation; the published package is the dependency. | Application fixture owns the 212-line handwritten-socket-io.js implementation. |
| Shared branches / workarounds | None in shared app, scenario, or assertions | None in shared app, scenario, or assertions | None in shared app, scenario, or assertions |
| Explicit failure / debugging surface | Observed source paths: listen errors, missing TCP address, and close callback errors are explicit. | Observed source path: this bootstrap adds no target-specific error branch. | Observed source paths: missing registered server and disconnected emitWithAck reject explicitly. |
| Locations changed with wiring / semantics | Dependency manifest/lock and fixture bootstrap when target wiring changes. | Dependency manifest/lock and fixture bootstrap when target wiring changes. | Fixture bootstrap plus handwritten-socket-io.js when exercised mock semantics change. |
| Directly observed simpler aspect | Reference behavior without application-owned mock logic. | Avoids HTTP server, ephemeral port, and explicit client activation setup. | Simplest dependency installation and port setup of the three recorded fixtures. |
Inference Because the handwritten fixture owns the observed 212-line implementation, changes to exercised room or event semantics may require changes there. Future maintenance effort was not measured.
03
Pinned implementation evidence
Selecting an approach changes its setup, owned files, failure paths, and exact source.
Real Socket.IO
socket.io@4.8.3, socket.io-client@4.8.3
Local HTTP server, real Socket.IO server/client, and an ephemeral loopback TCP port.
bootstrap.js· bootstrap · 61 lines
Observed source paths: listen errors, missing TCP address, and close callback errors are explicit.
case-studies/chat-room/fixtures/socket-io/bootstrap.js:1–61HTTP server, ephemeral port, client activation, and shutdown.
Open pinned source ↗import { createServer } from 'node:http';
import { Server } from 'socket.io';
import { io as connect } from 'socket.io-client';
import { createChatApplication } from './app.js';
import { runChatRoomScenario } from './scenario.js';
function listen(httpServer) {
return new Promise((resolve, reject) => {
httpServer.once('error', reject);
httpServer.listen(0, '127.0.0.1', () => {
httpServer.off('error', reject);
resolve();
});
});
}
function close(io) {
return new Promise((resolve, reject) => {
io.close((error) => {
if (error) reject(error);
else resolve();
});
});
}
function createClient(url, options) {
const client = connect(url, {
...options,
autoConnect: false,
forceNew: true,
reconnection: false,
});
return {
client,
activate: () => client.connect(),
};
}
export function runScenario() {
return runChatRoomScenario({
createClient,
async startApplication() {
const httpServer = createServer();
const io = new Server(httpServer);
await listen(httpServer);
const address = httpServer.address();
if (!address || typeof address === 'string') {
throw new Error('Socket.IO fixture did not receive a TCP address');
}
const url = `http://127.0.0.1:${address.port}`;
return createChatApplication({
io,
url,
close: () => close(io),
});
},
});
}04
Workflow behavior matrix
The same structured observation passed the same assertion for every target.
| Selected behavior | Real Socket.IO | Exact published Smocket | Handwritten mock |
|---|---|---|---|
| ✓ Passed · same observation | ✓ Passed · same observation | ✓ Passed · same observation | |
| ✓ Passed · same observation | ✓ Passed · same observation | ✓ Passed · same observation | |
| ✓ Passed · same observation | ✓ Passed · same observation | ✓ Passed · same observation | |
| ✓ Passed · same observation | ✓ Passed · same observation | ✓ Passed · same observation | |
| ✓ Passed · same observation | ✓ Passed · same observation | ✓ Passed · same observation | |
| ✓ Passed · same observation | ✓ Passed · same observation | ✓ Passed · same observation |
Selected structured observation · Acknowledged joins
[
{
"participantId": "alice",
"channel": "general",
"acknowledgement": {
"accepted": true,
"channel": "general"
}
},
{
"participantId": "alice",
"channel": "support",
"acknowledgement": {
"accepted": true,
"channel": "support"
}
},
{
"participantId": "bob",
"channel": "general",
"acknowledgement": {
"accepted": true,
"channel": "general"
}
},
{
"participantId": "carol",
"channel": "support",
"acknowledgement": {
"accepted": true,
"channel": "support"
}
}
]examples/chat-room/assertions.js:9–31Shared expected value used by all three targets.
Open pinned source ↗export const expectedObservation = {
joins: [
{
participantId: 'alice',
channel: 'general',
acknowledgement: { accepted: true, channel: 'general' },
},
{
participantId: 'alice',
channel: 'support',
acknowledgement: { accepted: true, channel: 'support' },
},
{
participantId: 'bob',
channel: 'general',
acknowledgement: { accepted: true, channel: 'general' },
},
{
participantId: 'carol',
channel: 'support',
acknowledgement: { accepted: true, channel: 'support' },
},
],examples/chat-room/app.js:20–32Shared application code used without a target branch.
Open pinned source ↗ socket.on('join-channel', async (channel, acknowledge) => {
if (!channels.has(channel)) {
acknowledge({ accepted: false, reason: 'unknown-channel' });
return;
}
await socket.join(channel);
io.to(socket.id).emit('welcome', {
channel,
text: `Welcome to #${channel}.`,
});
acknowledge({ accepted: true, channel });
});case-studies/chat-room/fixtures/handwritten/handwritten-socket-io.js:33–82Room union routing, deduplication, joins, and sender exclusion.
Open pinned source ↗class BroadcastOperator {
constructor(server, rooms, excludedSocketIds = new Set()) {
this.server = server;
this.rooms = new Set(rooms);
this.excludedSocketIds = excludedSocketIds;
}
emit(event, ...args) {
const recipients = this.rooms.size === 0 ? new Set(this.server.sockets.keys()) : new Set();
for (const room of this.rooms) {
for (const socketId of this.server.rooms.get(room) ?? []) {
recipients.add(socketId);
}
}
for (const socketId of recipients) {
if (this.excludedSocketIds.has(socketId)) continue;
this.server.sockets.get(socketId)?.deliver(event, cloneArguments(args));
}
return true;
}
}
class ServerSocket extends Emitter {
constructor(id, server, client, auth) {
super();
this.id = id;
this.server = server;
this.client = client;
this.handshake = { auth: auth ?? {} };
this.rooms = new Set();
this.connected = true;
}
join(roomOrRooms) {
if (!this.connected) return;
for (const room of asArray(roomOrRooms)) {
this.rooms.add(room);
const members = this.server.rooms.get(room) ?? new Set();
members.add(this.id);
this.server.rooms.set(room, members);
}
}
to(roomOrRooms) {
return new BroadcastOperator(this.server, asArray(roomOrRooms), new Set([this.id]));
}05
Supporting observation evidence
The shared transcript is secondary because target selection cannot change it.
Supporting evidence: shared transcript 10 canonical lines
Filter transcript by participant
Filter transcript by event
Showing 10 of 10 shared lines.
- 01
[alice] Welcome to #general. - 02
[alice] Welcome to #support. - 03
[bob] Welcome to #general. - 04
[carol] Welcome to #support. - 05
[alice] Bob in #general: Hello, everyone! - 06
[bob] Announcement rejected: moderator-only - 07
[alice] Alice to #general, #support: Maintenance starts at 18:00. - 08
[bob] Alice to #general, #support: Maintenance starts at 18:00. - 09
[carol] Alice to #general, #support: Maintenance starts at 18:00. - 10
[alice] Bob left #general.
06
Interpretation & evidence boundaries
Three questions require three different limits on what this record can support.
Fidelity
Published Smocket did not change the selected application's observable result relative to real Socket.IO. This says nothing about behavior outside the shared assertions; the conformance report remains authoritative for declared compatibility.
Reliability
The runner is repeatable and each recorded target passed twice in one process. This is one snapshot, not evidence of continued success over time. Recurring published-package validation is separate integration evidence.
Productivity
Physical source lines, including blank and comment lines, describe concrete code surfaces. They are not a productivity score, and generated lockfiles are excluded from that comparison.
Neutral and unfavorable findings
- The handwritten target is simpler in dependency installation and port setup.
- The real target supplies reference behavior without application-owned mock logic.
- Published Smocket keeps the shared application and assertions unchanged with an in-memory bootstrap.
Inferencechanges to the exercised event or room semantics may require maintaining the handwritten mock's additional implementation surface.
07
Reproduction & provenance
Run these commands in the pinned Smocket source. Recording replaces the canonical snapshot, so use it only intentionally.
Commands
pnpm case-study:chat-roompnpm case-study:chat-room:checkpnpm case-study:chat-room:recordnode scripts/run-chat-room-case-study.mjs --target socket-ionode scripts/run-chat-room-case-study.mjs --target published-smocketnode scripts/run-chat-room-case-study.mjs --target handwritten
Shared application source
examples/chat-room
app.jsapplication · 86 physical source linesscenario.jsapplication · 183 physical source linesassertions.jsassertions · 98 physical source lines
Recorded provenance
- Recorded at
2026-08-12T08:51:16.147Z- Environment
darwin arm64; Node v22.16.0; npm 11.17.0- Pinned source commit
fa90e07e272c7fd0db64ebfd73cbb104664ddb81- Authoritative publication commit
6a17477beef33fb014ab629b914d80a6f144b31b- Observation SHA-256
414b07fb27b70cc836d8b71d78d63a0f530d2cae28dbd32b60e77462a64f4bad- Application source SHA-256
e3884c42af5987b4db154c7f13538054e405e12b496803b8d321ac9a409b62d5