nsaspy/starintel-server
1
0
Fork 0
Code Issues 49 Pull requests 7 Projects Releases Packages Wiki Activity Actions

security audit and OWASP ZAP CI #105

Open
lost-rob0t wants to merge 43 commits from agent/security-audit-zap-ci into master
pull from: agent/security-audit-zap-ci
merge into: nsaspy:master
Conversation 3 Commits 43 Files changed 16 +2291 -53
lost-rob0t commented 2026-08-19 15:05:38 +00:00 (Migrated from github.com)
Copy link

What changed

  • remove the implicit first-run star:intel administrator credential
  • leave the human-user store empty unless an initial password is explicitly configured
  • preserve the one-time bootstrap-secret API-key path for fresh deployments
  • enforce credential grant attenuation so delegated issuers cannot mint admin or broader-scope credentials
  • bind non-admin delegated credentials to the caller's own principal ID and principal type, preventing cross-principal impersonation
  • prevent delegated rotate/revoke/disable operations from crossing identity or targeting credentials more authoritative than the caller
  • make administrator authority depend on explicit admin scope, not the descriptive principal-type string
  • explicitly map human-user management and self-password-change routes through the centralized authorization boundary
  • add global HTTP response hardening headers
  • add an OWASP ZAP Automation Framework scan against a disposable Compose stack
  • add Common Lisp reader/eval and process-execution RCE canaries that fail if a /tmp marker is created inside the disposable server
  • add CouchDB/Clouseau injection probes for boolean/field scope escape and encoded design/view path traversal
  • add malformed JSON, invalid UTF-8, duplicate-key observations, oversized-body, deep-nesting, bulk-amplification, and bounded concurrent parser tests
  • hard-refuse the negative-test script unless it is running inside a starintel-zap-* disposable Compose project with STAR_SECURITY_DISPOSABLE=1
  • upload ZAP reports, negative-test JSONL results, CouchDB/server logs, and stack diagnostics as workflow artifacts
  • document the security audit, hostile-input suite, residual findings, and next hardening priorities

Security rationale

The audit found four privilege-boundary failures:

  1. A fresh auth database created an administrator account with a known built-in password. The public login route could exchange that password for a normal administrator API key before the required-password-change flag had any enforcement effect.
  2. A principal with only credentials:create could request arbitrary scopes and principal type, including an administrator credential with admin. Lifecycle capabilities could also rotate a more privileged credential and receive its replacement secret.
  3. API-key owner becomes the authenticated principal ID, but delegated creation accepted any owner. A credential issuer could therefore mint a same-scope key as another principal and bypass identity/ownership checks. Lifecycle operations had the same cross-identity problem.
  4. administrator-principal-p treated the descriptive principal type administrator as authority even without admin scope, while the central policy engine used scope-based authority. The bulk-job status path used that helper for its ownership bypass.

This branch removes those privilege paths and now actively tests hostile input against the deployed CI image. Lisp injection probes are non-destructive canaries: successful execution only writes a marker under /tmp, which is still treated as proof of RCE. CouchDB tests use two datasets with the same search fixture and fail if a restricted principal can retrieve the denied document through Lucene/Clouseau syntax manipulation. DoS probes are deliberately bounded and run only against the disposable stack.

The remaining audit findings are deliberately documented rather than hidden: direct HTTP bearer transport has no in-process TLS invariant, must_change_password is still advisory, development auth bypass needs a loopback invariant, the application body limit is downstream of HTTP request materialization, and container/CI least privilege can be tightened further.

Validation

GitHub CI is the validation environment for this branch. Existing Nix/unit/integration/container jobs plus the ZAP and hostile-input workflow must pass before this leaves draft.

## What changed - remove the implicit first-run `star:intel` administrator credential - leave the human-user store empty unless an initial password is explicitly configured - preserve the one-time bootstrap-secret API-key path for fresh deployments - enforce credential grant attenuation so delegated issuers cannot mint `admin` or broader-scope credentials - bind non-admin delegated credentials to the caller's own principal ID and principal type, preventing cross-principal impersonation - prevent delegated rotate/revoke/disable operations from crossing identity or targeting credentials more authoritative than the caller - make administrator authority depend on explicit `admin` scope, not the descriptive principal-type string - explicitly map human-user management and self-password-change routes through the centralized authorization boundary - add global HTTP response hardening headers - add an OWASP ZAP Automation Framework scan against a disposable Compose stack - add Common Lisp reader/eval and process-execution RCE canaries that fail if a `/tmp` marker is created inside the disposable server - add CouchDB/Clouseau injection probes for boolean/field scope escape and encoded design/view path traversal - add malformed JSON, invalid UTF-8, duplicate-key observations, oversized-body, deep-nesting, bulk-amplification, and bounded concurrent parser tests - hard-refuse the negative-test script unless it is running inside a `starintel-zap-*` disposable Compose project with `STAR_SECURITY_DISPOSABLE=1` - upload ZAP reports, negative-test JSONL results, CouchDB/server logs, and stack diagnostics as workflow artifacts - document the security audit, hostile-input suite, residual findings, and next hardening priorities ## Security rationale The audit found four privilege-boundary failures: 1. A fresh auth database created an administrator account with a known built-in password. The public login route could exchange that password for a normal administrator API key before the required-password-change flag had any enforcement effect. 2. A principal with only `credentials:create` could request arbitrary scopes and principal type, including an `administrator` credential with `admin`. Lifecycle capabilities could also rotate a more privileged credential and receive its replacement secret. 3. API-key `owner` becomes the authenticated principal ID, but delegated creation accepted any owner. A credential issuer could therefore mint a same-scope key as another principal and bypass identity/ownership checks. Lifecycle operations had the same cross-identity problem. 4. `administrator-principal-p` treated the descriptive principal type `administrator` as authority even without `admin` scope, while the central policy engine used scope-based authority. The bulk-job status path used that helper for its ownership bypass. This branch removes those privilege paths and now actively tests hostile input against the deployed CI image. Lisp injection probes are non-destructive canaries: successful execution only writes a marker under `/tmp`, which is still treated as proof of RCE. CouchDB tests use two datasets with the same search fixture and fail if a restricted principal can retrieve the denied document through Lucene/Clouseau syntax manipulation. DoS probes are deliberately bounded and run only against the disposable stack. The remaining audit findings are deliberately documented rather than hidden: direct HTTP bearer transport has no in-process TLS invariant, `must_change_password` is still advisory, development auth bypass needs a loopback invariant, the application body limit is downstream of HTTP request materialization, and container/CI least privilege can be tightened further. ## Validation GitHub CI is the validation environment for this branch. Existing Nix/unit/integration/container jobs plus the ZAP and hostile-input workflow must pass before this leaves draft.
lost-rob0t commented 2026-08-24 05:21:52 +00:00 (Migrated from github.com)
Copy link

RAGE/TDD failure evidence for the current repair transaction:

  • immutable failing head: 055c8bfdb5f291c764decd676c2163911e851021
  • failing command: bash ./scripts/zap-ci.sh passive
  • deterministic behavior: OWASP ZAP Security failed while establishing the scoped CouchDB/Clouseau fixtures; POST /new/document/note returned HTTP 422 before the hostile search probes could run
  • uploaded server diagnostics identified the exact server response as unsupported_schema_version: Document schema version is not supported
  • cause: scripts/security-negative-test.sh hard-coded document version:"0.9.0", so the security harness had drifted from the server's canonical document spec
  • proof: the failure occurs in fixture creation, before the assertions intended to detect Clouseau/CouchDB scope escape; therefore weakening those assertions would be a false green

Repair commit c7d01c49538e5ab980cf3a9c3af0266e984d888d now discovers doc_spec_version from the server metadata endpoint and uses that value for all canonical document fixtures, while retaining the hostile-input assertions. It also emits the response body on fixture creation failure instead of hiding the reason behind curl --fail.

The branch then conflicted with #104/master, so current master 8f6e50701081f0335c7552082dba5ff392b859e5 was merged explicitly. Current exact head is 1c6f1efacc531d67a7d86a9dd29ee5d414bad38f; it is 0 commits behind master and preserves #104's client/admin systems/tests plus #105's security hardening. Fresh exact-head CI is in progress; do not merge or remove draft status until every required gate, especially ZAP, is green.

RAGE/TDD failure evidence for the current repair transaction: - immutable failing head: `055c8bfdb5f291c764decd676c2163911e851021` - failing command: `bash ./scripts/zap-ci.sh passive` - deterministic behavior: OWASP ZAP Security failed while establishing the scoped CouchDB/Clouseau fixtures; `POST /new/document/note` returned HTTP 422 before the hostile search probes could run - uploaded server diagnostics identified the exact server response as `unsupported_schema_version: Document schema version is not supported` - cause: `scripts/security-negative-test.sh` hard-coded document `version:"0.9.0"`, so the security harness had drifted from the server's canonical document spec - proof: the failure occurs in fixture creation, before the assertions intended to detect Clouseau/CouchDB scope escape; therefore weakening those assertions would be a false green Repair commit `c7d01c49538e5ab980cf3a9c3af0266e984d888d` now discovers `doc_spec_version` from the server metadata endpoint and uses that value for all canonical document fixtures, while retaining the hostile-input assertions. It also emits the response body on fixture creation failure instead of hiding the reason behind `curl --fail`. The branch then conflicted with #104/master, so current master `8f6e50701081f0335c7552082dba5ff392b859e5` was merged explicitly. Current exact head is `1c6f1efacc531d67a7d86a9dd29ee5d414bad38f`; it is 0 commits behind master and preserves #104's client/admin systems/tests plus #105's security hardening. Fresh exact-head CI is in progress; do not merge or remove draft status until every required gate, especially ZAP, is green.
lost-rob0t commented 2026-08-24 05:31:34 +00:00 (Migrated from github.com)
Copy link

Second-stage TDD evidence: the stale-schema harness defect is fixed, and the hostile suite now exposes a real server boundary defect.

At exact branch head 6cba035145d7a4b5b8c35d77a64596e4f3494f3b (PR merge ref 83f7875de3be3314e9be79b4459a01593e1b7db4), OWASP ZAP Security run 32693397947, job 97331044353, reached the malformed-JSON phase after stack health, delegated credential checks, scoped CouchDB/Clouseau fixtures, search injection, and view-path probes all succeeded.

The deterministic regression is now:

malformed JSON probe trailing-comma expected 400, got HTTP 200

The request used the valid canonical document schema version 0.8.0; the server accepted and normalized it as {"_id":"security-json-comma","dataset":"security-a","dtype":"note","version":"0.8.0"}. This proves malformed JSON is reaching successful document handling rather than being rejected at the HTTP syntax boundary.

Per the StarIntel human-gated ADADR policy, no parser implementation is being added to this PR yet. The research transaction is now lost-rob0t/starintel-auto-research#142 / STAR-RESEARCH-041 Strict JSON HTTP Boundary, with research approval PENDING, architecture NOT STARTED, and implementation BLOCKED until explicit operator research approval, subsequent design/adversarial review, and separate design approval.

Keep this PR draft/red. Do not weaken the expected 400 regression to make ZAP green.

Second-stage TDD evidence: the stale-schema harness defect is fixed, and the hostile suite now exposes a real server boundary defect. At exact branch head `6cba035145d7a4b5b8c35d77a64596e4f3494f3b` (PR merge ref `83f7875de3be3314e9be79b4459a01593e1b7db4`), OWASP ZAP Security run `32693397947`, job `97331044353`, reached the malformed-JSON phase after stack health, delegated credential checks, scoped CouchDB/Clouseau fixtures, search injection, and view-path probes all succeeded. The deterministic regression is now: `malformed JSON probe trailing-comma expected 400, got HTTP 200` The request used the valid canonical document schema version `0.8.0`; the server accepted and normalized it as `{"_id":"security-json-comma","dataset":"security-a","dtype":"note","version":"0.8.0"}`. This proves malformed JSON is reaching successful document handling rather than being rejected at the HTTP syntax boundary. Per the StarIntel human-gated ADADR policy, no parser implementation is being added to this PR yet. The research transaction is now `lost-rob0t/starintel-auto-research#142` / `STAR-RESEARCH-041 Strict JSON HTTP Boundary`, with research approval PENDING, architecture NOT STARTED, and implementation BLOCKED until explicit operator research approval, subsequent design/adversarial review, and separate design approval. Keep this PR draft/red. Do not weaken the `expected 400` regression to make ZAP green.
lost-rob0t commented 2026-08-25 22:11:39 +00:00 (Migrated from github.com)
Copy link

RAGE freshness reconciliation (2026-08-25): the trailing-comma RED evidence on this PR is no longer sufficient to establish a defect on current master.

The immutable failure remains valid for head 6cba035145d7a4b5b8c35d77a64596e4f3494f3b / ZAP run 32693397947, but current master is now 52948721d8981e9aa5ccad4efd77185e52af845b after #119. This branch has diverged substantially from that master (merge base 8f6e50701081f0335c7552082dba5ff392b859e5; current-master side is 74 commits beyond the #105 head in the compare).

More importantly, current master changed the HTTP parser boundary: source/frontends/http-boundary-core.lisp now decodes UTF-8, validates the raw JSON text with yason:parse, and only then converts it to the existing JSOWN representation. That is directly relevant to the old failure mode where permissive JSOWN parsing accepted a trailing comma. Current master also has a malformed-json-is-a-400-client-error unit regression, although it currently uses a truncated { fixture rather than the exact trailing-comma payload.

Therefore do not implement a parser fix from the old RED badge and do not merge/rebase this security branch wholesale. The next TDD action is to re-prove the exact trailing-comma regression against current master 52948721... (preferably the smallest parse-json-octets fixture plus the deployed HTTP/ZAP probe). If current master returns malformed_json/HTTP 400, record #119 as observed superseding implementation evidence and reconcile STAR-RESEARCH-041 without fabricating approval. If it still returns 200, preserve the RED evidence and continue through the existing human-gated ADADR transaction.

RAGE freshness reconciliation (2026-08-25): the trailing-comma RED evidence on this PR is no longer sufficient to establish a defect on current `master`. The immutable failure remains valid for head `6cba035145d7a4b5b8c35d77a64596e4f3494f3b` / ZAP run `32693397947`, but current `master` is now `52948721d8981e9aa5ccad4efd77185e52af845b` after #119. This branch has diverged substantially from that master (merge base `8f6e50701081f0335c7552082dba5ff392b859e5`; current-master side is 74 commits beyond the #105 head in the compare). More importantly, current master changed the HTTP parser boundary: `source/frontends/http-boundary-core.lisp` now decodes UTF-8, validates the raw JSON text with `yason:parse`, and only then converts it to the existing JSOWN representation. That is directly relevant to the old failure mode where permissive JSOWN parsing accepted a trailing comma. Current master also has a `malformed-json-is-a-400-client-error` unit regression, although it currently uses a truncated `{` fixture rather than the exact trailing-comma payload. Therefore do **not** implement a parser fix from the old RED badge and do not merge/rebase this security branch wholesale. The next TDD action is to re-prove the *exact* trailing-comma regression against current master `52948721...` (preferably the smallest `parse-json-octets` fixture plus the deployed HTTP/ZAP probe). If current master returns `malformed_json`/HTTP 400, record #119 as observed superseding implementation evidence and reconcile STAR-RESEARCH-041 without fabricating approval. If it still returns 200, preserve the RED evidence and continue through the existing human-gated ADADR transaction.
nsaspy changed title from WIP: security audit and OWASP ZAP CI to security audit and OWASP ZAP CI 2026-08-28 03:29:47 +00:00
This pull request has changes conflicting with the target branch.
  • doc/security-audit-2026-08-19.org
  • doc/security-negative-testing.org
  • source/gserver-settings.lisp
  • starintel-gserver-tests.asd
View command line instructions

Manual merge helper

Use this merge commit message when completing the merge manually.

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin agent/security-audit-zap-ci:agent/security-audit-zap-ci
git switch agent/security-audit-zap-ci

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch master
git merge --no-ff agent/security-audit-zap-ci
git switch agent/security-audit-zap-ci
git rebase master
git switch master
git merge --ff-only agent/security-audit-zap-ci
git switch agent/security-audit-zap-ci
git rebase master
git switch master
git merge --no-ff agent/security-audit-zap-ci
git switch master
git merge --squash agent/security-audit-zap-ci
git switch master
git merge --ff-only agent/security-audit-zap-ci
git switch master
git merge agent/security-audit-zap-ci
git push origin master
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
bug
Something isn't working

  
documentation
Improvements or additions to documentation

  
duplicate
This issue or pull request already exists

  
enhancement
New feature or request

  
good first issue
Good for newcomers

  
help wanted
Extra attention is needed

  
invalid
This doesn't seem right

  
question
Further information is requested

  
wontfix
This will not be worked on

No labels
bug
documentation
duplicate
enhancement
good first issue
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
nsaspy/starintel-server!105
No description provided.