Pulse_Intel

Pulse Intelligence — Handover

Last updated: 2026-07-23 Status: Phases 1–7 of 8 complete. 143 tests, clean build, 38 routes. Now running on real PostgreSQL 17 (§4.1 resolved). CVE data backfilled current as of 2026-07-23; see §4.5.

A self-hosted Threat Intelligence Platform — APT tracking, campaigns, IOC management with bulk enrichment, MITRE ATT&CK, automated feed ingestion, search, and threat hunting.

1. Moving to the RDP box — read this first

Two things will bite you, in this order.

1.1 The repo has no remote

git remote -v is empty. The code exists only in this folder until a GitHub remote is added.

For the open-source community release, create or choose the public GitHub repository, then:

cd "C:\Users\Sukesh\Downloads\Pulse Development\pulse_intelligence"
git remote add origin https://github.com/<you>/pulse-intelligence.git
git push -u origin main

Then on any new machine: git clone https://github.com/<you>/pulse-intelligence.git

Do not commit .env, real provider keys, database dumps, or private investigation data. The application source is safe to publish; deployment state is not.

Option B — copy the folder. Copy pulse_intelligence\ including the .git directory (700 KB — the whole history comes with it). Do not copy node_modules or .next; run npm install on the other side instead.

1.2 Secrets do NOT travel with git

.env is gitignored — correctly, but it means a fresh clone has no API keys and no database URL. You must recreate .env on the RDP box by hand. .env.example is committed as the template; the real values are only in your local .env.

Copy it across out-of-band (password manager, not email/chat), or regenerate the keys.

Rotate the three API keys. VirusTotal, AbuseIPDB and OTX keys were pasted into a chat transcript on 2026-07-21. Nothing is known to be compromised, but they have been seen by more than one system. Free-tier keys are trivial to regenerate: VirusTotal · AbuseIPDB · OTX

2. Bringing it up on a new machine

2.1 Prerequisites

Requirement Notes
Node.js 20.9+ Next.js 16 minimum. Developed on 25.8.1.
PostgreSQL 17 winget install PostgreSQL.PostgreSQL.17 — native Windows, no virtualization needed. See §4.1 before using npm run db:dev instead.
Redis-compatible server Memurai on the old box (winget install Memurai.MemuraiDeveloper). Needed for enrichment + scheduling.
Docker / WSL2 Not required. The old machine had the hypervisor disabled and everything still worked.

2.2 First run

npm install

# create .env from .env.example, fill in DATABASE_URL, REDIS_URL and the three API keys

npm run db:migrate          # real Postgres. (dev server: see §4.1)
npm run db:seed             # 3 users — admin/analyst/viewer @pulse.local
npm run db:seed:demo        # optional: 8 real APTs, 4 campaigns, sample IOCs
npm run attack:sync -- --all  # MITRE ATT&CK v19.1 (~58 MB download, ~1 min)
npm run feeds:install       # register the 18-source feed catalogue

# then, in two terminals:
npm run dev                 # http://localhost:3000
npm run worker -- --run-now # feeds + enrichment; --run-now pulls everything immediately

Sign in as admin@pulse.local / PulseAdmin!2026. Change these before the app is reachable by anyone else (SEED_PASSWORD overrides the seed default).

2.3 Sanity checks

npm run test                          # 96 tests, all pure logic — no DB or network
npm run typecheck
npm run verify:enrichment -- --live   # proves the rate limiter + all three API keys
npm run build

verify:enrichment is the highest-value one: it fires 20 concurrent requests at a 4/min quota and asserts exactly 4 are allowed. If that fails, Redis or the limiter is broken.

2.4 Sharing a public link (temporary — set up 2026-07-23)

For letting people outside this machine reach it: run production, not dev (npm run build && npm run start — dev mode is fine for local iteration, not for anyone else’s traffic), then a Cloudflare quick tunnel: cloudflared.exe tunnel --url http://localhost:3000 (binary downloaded to %TEMP%/installers this session; grab a fresh copy from https://github.com/cloudflare/cloudflared/releases/latest if it’s gone). No account, no config, no firewall/router changes — it’s an outbound-only connection to Cloudflare’s edge.

The URL is ephemeral. It’s tied to the cloudflared process; killing it (or a machine restart) kills the URL, and a new run gets a new random one — there’s no way to keep the same link across restarts without a Cloudflare account and a named tunnel. Treat any link handed out this way as short-lived.

Known risk, accepted, not fixed: the seed admin/analyst/viewer passwords are still the defaults printed in this file (§ below) and this repo’s own commit history/chat — anyone who has seen them can log in as ADMIN through the public link. There is no self-service change-password UI yet. If this stops being “show a few friends” and becomes anything longer- lived, rotate SEED_PASSWORD and manually update the three User.passwordHash rows (re-running db:seed won’t touch an existing user’s password — see the comment in prisma/seed.ts).

3. Where things are

prisma/schema.prisma           the data model — read this first
prisma/migrations/             hand-managed; see §4.2
src/
  proxy.ts                     renamed from middleware.ts in Next 16. Cookie check only.
  worker.ts                    background process: feeds, enrichment, scheduling
  lib/
    db.ts                      Prisma client (driver adapter — Prisma 7 requires one)
    redis.ts                   shared Redis + BullMQ connection factory
    auth/dal.ts                THE auth gate. requireUser / requireRole.
    actions.ts                 withAction() — RBAC + zod + error handling for mutations
    ioc/normalize.ts           refang, type detection, dedup normalization (43 tests)
    ioc/whitelist.ts           never-block list (9 tests)
    ioc/ingest.ts              the single ingest path for ALL indicator sources
    attack/stix.ts             ATT&CK STIX parsing (16 tests)
    attack/sync.ts             ATT&CK ingest
    enrichment/limiter.ts      Redis token bucket (the Lua script)
    enrichment/registry.ts     provider order — OTX first, VirusTotal last
    enrichment/providers/      virustotal, abuseipdb, otx, stub
    feeds/parsers.ts           RSS/CSV/KEV/EPSS/NVD parsing (20 tests)
    feeds/catalog.ts           the 18 preconfigured sources
    feeds/run.ts               feed execution + health tracking
    queue/queues.ts            BullMQ queues and priorities
  app/(app)/                   authenticated pages
  app/(auth)/                  login
scripts/                       CLI: attack-sync, install-feeds, migrate-offline, cve-catchup,
                                verify-enrichment/hunting/api/reports

4. Known issues and traps

4.1 RESOLVED — now on real PostgreSQL 17

As of 2026-07-22 this runs against a native PostgreSQL 17 service on the RDP box (postgresql-x64-17, port 5432), not the old prisma dev toy. Databases pulse and pulse_shadow exist; .env points at them. The old wedging / P1017 / dynamic-upsert problems are gone.

Two things still bite, both worked around:

Still open (low priority): the sequential-query comments in src/app/(app)/page.tsx and enrichment/page.tsx were written for the toy DB’s concurrency limits and can go back to Promise.all on real Postgres.

4.2 Migrations will try to delete your search indexes

Every prisma migrate diff emits DROP INDEX for the 13 pg_trgm/FTS indexes created in 20260721010000_search_indexes, because they are raw SQL and don’t exist in schema.prisma. Applying that silently turns fuzzy and full-text search into sequential scans — no error, just slow.

scripts/migrate-offline.ts refuses such a migration and prints the SQL to hand-edit. If you switch to prisma migrate dev, you lose that guard — check generated migrations for DROP INDEX yourself, every time.

4.3 Smaller traps that cost time

4.5 NVD needs an API key or CVE data quietly goes stale

The hourly “NVD Recent CVEs” feed (src/lib/feeds/run.ts, nvd-recent handler) only queries a rolling 24h lastModStartDate window — correct for staying current, useless for catching up. Without NVD_API_KEY set, NVD caps requests at 5/rolling-30s and intermittently 403s keyless traffic; the feed can report lastStatus: ok while genuinely making no progress, which is how the CVE table drifted to being ~7 weeks stale before anyone noticed.

Fixed 2026-07-23: NVD_API_KEY is now in .env (get one free, instantly, at https://nvd.nist.gov/developers/request-an-api-key), sent via the apiKey header (nvdHeaders() in run.ts). npm run cve:catchup [-- --days N] is a one-time backfill — unlike the hourly job, it pages through everything NVD published in the last N days (default 30), so a stale DB catches up in one run instead of trickling in 24 hours at a time. Ran once already: pulled 9,697 CVEs from the last 30 days. Re-run it (with a larger --days) any time the data looks stale again; the hourly job alone is only good at staying current once it is current.

4.6 Vulnerability retention — strict 90-day window, including KEV

The NVD feed pulls in the full public CVE stream, which grows unbounded with nothing to prune it. pruneOldVulnerabilities() (src/lib/feeds/run.ts) now runs automatically as part of every hourly nvd-recent job: deletes every vulnerability whose publishedAt is older than VULN_RETENTION_DAYS (90 — “last 2-3 months,” a product decision, not a technical constraint) — including CISA KEV entries, and including rows with no publishedAt at all (treated as unknown-age, pruned rather than kept forever).

This was not the first cut. The first version exempted KEV rows on the reasoning that “actively exploited” stays relevant regardless of age (Log4Shell is still on CISA’s list). That’s true, but the product call landed the other way — a self-hosted single-tenant instance would rather see a short, current list than carry CVE-2021-44228 forever, and the default EPSS-first sort meant KEV rows dominated page 1 regardless of the retention window, which read as “still showing 2021-2024 data” even after pruning. A second attempt fell back to kevDateAdded for undated KEV rows (when CISA flagged it, not when NVD published it) — that let CVE-2008-4250 stay on a “last 3 months” list because it was re-flagged recently, which is exactly the stale-looking noise this was supposed to remove. Landed on: judge age by publishedAt only, no exemptions, no fallback. If a KEV exemption is wanted back, gate the deleteMany on knownExploited: false — but expect the “page 1 still looks old” complaint to return with it, because the EPSS-first sort will keep surfacing whatever KEV rows survive.

Manual/immediate cleanup: npm run vuln:prune [-- --days N]. As of 2026-07-23: 10,608 CVEs (down from a peak of 12,345), oldest publishedAt in the table is ~90 days out, 15 CISA KEV entries remain (all recently published, not just recently flagged).

4.7 <body> needs suppressHydrationWarning

Browser extensions (Grammarly, password managers) inject data-* attributes onto <body> before React hydrates, which React reports as a hydration mismatch — a false positive; the extension modified the DOM, the app didn’t do anything wrong. suppressHydrationWarning on the root layout’s <body> (src/app/layout.tsx) is the standard, documented fix. Don’t add it anywhere deeper in the tree — it silences real mismatches too, and <body> is the only element extensions commonly touch.

4.8 Vulnerabilities page sorts newest-first; campaigns auto-link to news like actors do

/vulnerabilities now orders by publishedAt desc first, EPSS/CVSS as the tiebreaker (was the reverse — see §4.6 for why that read as stale). The dashboard’s separate “Highest-risk exploited CVEs” widget is untouched; it’s deliberately EPSS-first and correctly labeled as such.

NewsItem.linkedCampaignIds (new field, mirrors linkedActorIds/linkedCveIds) is populated by linkNewsItem() in src/lib/feeds/run.ts, now exported so scripts/relink-news.ts can also call it — that script re-scans every existing news item against the current actor/campaign roster, because a campaign added today never retroactively links to an article ingested yesterday otherwise (ingestNews only links at ingest time). Run npm run news:relink after adding campaigns or actors to the seed.

Matching is exact-name, word-boundary — same mechanism as actor linking, same limitation. “Scattered Spider” appears verbatim in real articles and matches fine; a constructed title like “MGM Resorts & Caesars Entertainment Ransomware Attacks” almost never will, because no journalist types the analyst-style campaign name verbatim. Actor linking works well in practice for this reason; campaign linking will mostly stay empty until real coverage happens to phrase it that way. This was a known, accepted tradeoff, not a bug — a fuzzy/keyword matcher would need a campaignAliases-style field (mirroring ActorAlias) to do meaningfully better, which wasn’t built. If this starts to matter, that’s the extension point.

prisma/seed-demo.ts grew from 4 to 15 campaigns — real, publicly-documented incidents (3CX, JumpCloud, MGM/Caesars, Snowflake, UK retail DragonForce attacks, WinRAR CVE-2023-38831, Industroyer2, KV Botnet, Black Basta, US state government zero-days, Teams device-code phishing), each attributed to one of the existing 8 actors. npm run db:seed:demo is idempotent (upsert on name) — safe to re-run after adding more.

4.9 SOC dashboard — real charts, hand-rolled SVG, no chart library

The / dashboard (src/app/(app)/page.tsx) grew from stat tiles + two lists into a proper SOC-style overview: two 30-day trend lines (new indicators/day, new CVEs published/day — computed directly from Indicator.createdAt / Vulnerability.publishedAt, no separate snapshot table, so history exists from day one instead of only after a new table starts accumulating rows) plus four distribution bar charts (indicator severity, indicator types, vulnerability CVSS band, ATT&CK tactic coverage — the last one reuses getMatrix() from src/lib/attack/matrix.ts rather than a new query).

5. Design rules — please don’t break these

These are load-bearing. Each exists because the alternative produces a platform that lies.

  1. Every relation carries confidence + addedById. Attribution is opinion, not fact. The UI must always be able to answer “who claimed this, and how sure were they?”. Imported MITRE mappings deliberately have no addedById so they display as “MITRE ATT&CK”, not as an analyst’s judgement.
  2. Indicator is unique on (type, normalizedValue). Feeds overlap heavily; dedup at ingest is the only thing keeping the table meaningful. All ingest goes through src/lib/ioc/ingest.ts so dedup, whitelisting and confidence apply identically.
  3. Whitelisting is never bypassed. The 8.8.8.8-in-a-C2-feed case is real. Whitelisted IOCs are stored (so you can see the feed claimed it) but never exported, alerted on, or enriched.
  4. Quota limits are surfaced, never hidden. VirusTotal is 4/min and 500/day; a 10k-indicator batch takes ~20 days and /enrichment says so. Never show a spinner that implies “soon”.
  5. Provider order is OTX → AbuseIPDB → VirusTotal. Cheapest first; every OTX answer is a VirusTotal request preserved.
  6. Report bodies render as preformatted text, not HTML. They contain attacker-controlled strings; a markdown→HTML renderer would be an XSS sink.
  7. ATT&CK is version-pinned in src/lib/attack/stix.ts. MITRE reshapes fields between releases (v19 moved detection off the technique object entirely). Upgrading is a deliberate act, never something a cron does.
  8. proxy.ts only checks for a cookie. Real verification is in lib/auth/dal.ts, per Next’s guidance that proxy must not rely on shared modules. Don’t move auth into proxy.

6. Current data (as of handover)

Reproducible from scratch with the §2.2 commands.

   
Threat actors 8 real APTs, 30 cross-vendor aliases
Campaigns 15 real, publicly-documented campaigns (2017-2025), attributed to the 8 actors
ATT&CK 943 techniques, 41 tactics (v19.1), 634 MITRE group mappings
Vulnerabilities 10,608 CVEs — strict 90-day retention, no exemptions (§4.6), 15 CISA KEV, newest-first
Indicators ~2,000 from abuse.ch, OTX, and the demo seed
News 194 articles, auto-linked to actors, CVEs, and campaigns (exact-name match)
Feeds 18 sources, hourly

7. What’s next

Phase 6 — Threat hunting. ✅ done 2026-07-22. Structured query builder over the indicator set (/hunting), saved + scheduled hunts, and alerting on new matches.

Phase 7 — Export and API. ✅ done 2026-07-22.

Phase 7 is complete.

Phase 8 — Hardening. Indicator decay (the decayHalfLifeDays field is set per source but nothing applies it yet), Indicator partitioning if it passes ~10M rows, backup/restore, rate limits on the public API, security review.

Also outstanding:

This site is open source. Improve this page.