Schema-Based Testing Setup ¶

This article describes how to enable and configure Wallarm's Schema-Based Testing.

Enable¶

Schema-Based Testing is disabled by default. To enable:

1. Contact sales@wallarm.com to activate the Security Testing subscription plan for your account.

2. Go to the Security Testing → Schema-Based → Test policies tab and create at least one policy.

Prerequisites¶

Token¶

Schema-Based Testing requires a token for authorizing data exchange between running Docker container and Wallarm Cloud. The token can be created in two ways:

• Automatically - Schema-Based Testing will create it automatically and include into the docker run command on first attempt to copy Docker command from any policy. Other policies will re-use already existing token.

• Manually - in Wallarm Console, go to Settings → API Tokens, click New token; on creation, set Token usage to Schema-Based Testing agent. All policies will use this token.

Whitelist¶

When you are planning to run Wallarm's Schema-Based Testing against domains protected by some security tools, it is recommended to add the client IP (IP from where the Docker command is being run) to the allowlist, otherwise majority of requests will be blocked and the vulnerabilities will not be found.

This includes the case when Wallarm itself is used as the protection tool for these domains - see details on Wallarm's allowlist here.

Test policy types¶

You can configure test policy based on OpenAPI specification (OAS) or Postman collection. Note that one test policy is aimed at only one type of testing (either OAS or Postman-based).

• OAS is more focused on input validation, injection, misconfiguration detection

• Postman - on complex business logic and access control vulnerabilities

OAS-based test policies¶

OpenAPI specification (OAS)-based test policy defines persistently:

• Application's OpenAPI specification

• Tests to run

Besides persistent parameters that are the same for any test run, each test policy may optionally include parameters that can be overridden during each next test run (Runtime parameters). Re-defining the runtime parameters can be useful for embedding of Docker into the CI/CD pipelines:

• Application's Target URL

  (although can be redefined during each run, some initial value is required)

• Authentication parameters

To configure test policy:

1. Go to Wallarm Console → Security Testing → Schema-Based → Test policies.

2. Click Add policy, attach OpenAPI specification file.

3. Select test types to run.

4. Set Target URL (can be overridden dynamically during each test run).

5. Optionally, add authentication Runtime parameters.

   

Postman collection-based test policies¶

With Postman-based security testing you can automate security scans alongside your regular API tests, ensuring that each API run is thoroughly tested for vulnerabilities.

Pre-check of Postman collection¶

Before using Postman data for schema-based security testing:

1. Ensure that you collection and environment files contain:

   • Functional tests for API endpoints
   • Location of the target application
   • All environment variables set
   • Necessary credentials to authenticate in the target application

2. (Recommended) Check whether the Postman collection is working. For example, this can be done by running the command:

   newman run my_collection.json -e my_environment.json
   

   This can tell you in advance whether there are any problems, for example, related to the quality of the Postman collection, the availability of the target URL, or that all the necessary variables are specified.

Configuring test policy in Wallarm¶

In Wallarm, Postman collection-based test policy defines:

• Application's Postman collection.

• Postman environment file(s) (optional if all configuration is stored in the main collection file).

  Target URL and authentication

  Application's Target URL and authentication parameters are defined in the Postman collection or environment files.

• Test case selection is not currently supported for security testing based on Postman collections.

To configure test policy:

1. Go to Wallarm Console → Security Testing → Schema-Based → Test policies.

2. Click Add policy, set Source to Postman collection.

3. Attach Postman collection file.

4. Optionally, attach Postman environment file(s). Attaching 2 files allows running testing twice with different variable values (for example, different credentials) and then comparing results.

   

Business logic security testing¶

For business logic testing (OWASP API1:2023 Broken Object Level Authorization (BOLA) and OWASP API5:2023 Broken Function Level Authorization (BFLA)) Wallarm's Schema-Based Security Testing requires API traffic from at least two different authenticated users, preferably with varying privileges (e.g., admin and regular users). This diversity enables more effective business logic security tests and provides a more thorough assessment.

Example 1: Using Postman collections with requests from two users¶

Do the following:

1. Create a Postman collection containing requests from two authenticated users. For example, in the target application, include login and activity requests from User A and User B in the same collection.

   

2. Verify that all requests are executed correctly.

3. Create a test policy with the created Postman collection.

Example 2: Using Postman environments for multiple users¶

Do the following:

1. Create two Postman environment files, each holding the credentials for a different authenticated user, e.g.:

   • env1.json for User A
   • env2.json for User B

2. Create a test policy that uses your Postman collection and both environment files.

   In this setup, the collection is executed twice - once with env1.json (User A) and once with env2.json (User B) - so each request runs under both user contexts.

Editing existing policy¶

You can edit previously created policies: while clicking policy itself opens its Docker command info, you can click the edit button to access the edit dialog:

Docker run¶

Running with test policy¶

As test policy is created, it provides you with the Docker run command which allows you start tests for your application:

1. Go to Wallarm Console → Security Testing → Schema-Based → Test policies.

2. Click you policy. The policy's Docker command will be displayed.

3. If necessary, redefine Docker log level and format, and test run success criteria. This adds extra variables to the command.

   

4. Copy command and run it or embed into your CI/CD pipeline. This will run security tests selected in the policy for your application.

   Remember that, for OAS-based run, you can override the policy's Runtime parameters on each run by adding the corresponding -e parameters to the Docker run command, for example:

   -e TARGET_URL="http://dvapi.st.wallarm.tools" 
   -e AUTH_HEADER="Authorization: Bearer <VALUE>"
   

   You can simplify re-defining these parameters by selecting the Rewrite authentication data and/or Rewrite target URL checkboxes that add to the command:

   -e AUTH_HEADER="${AUTH_HEADER}" -e TARGET_URL="${TARGET_URL}"
   

5. View run statistics and test run results on the Test runs tab.

Running without test policy¶

Having token, you can form Docker run command yourself defining all parameters locally. Note that you can still use Wallarm's wizard to help you with that.

To use wizard:

1. Go to Schema-Based Testing section.

2. Click the Generate test run command button.

3. Select appropriate parameters, the wizard will compose the Docker run command correspondingly.

   

4. Copy the command.

5. Replace the placeholders with actual values.

6. You have the command.

Note that:

• If wizard is left with default values, it does not add anything additional to Docker command.

• Export input data to the Cloud if you use this, OAS or Postman collection file that you use locally to run tests, will be uploaded to Test runs for improved debugging if needed later.

Running combination¶

You create test policy in Wallarm that will contains token, your schema and all testing parameters and provide you a Docker run command linked to it, but in this linked command you override any parameters, like use local file instead of the one attached to the policy. What is not overridden, is taken from the policy.

See details on how to use the policy with local override in the Docker -help.

Environment variables¶

When you create and save a test policy in Wallarm, it automatically creates a Docker run command, adds all required -e variables and sets values for them. Generally, variables aim to tell Docker container to which account in Wallarm Cloud to connect and which test policy to take as the basis for testing of your application:

Available flags¶

You can use flags to modify behavior of Wallarm's Schema-Based Testing running as Docker container. To see the full list of available flags, use:

<YOUR DOCKER RUN COMMAND FOR SCHEMA-BASED TESTING> --help

The outup will be something like:

Run OpenAPI security testing workflow.

This command performs automated API security testing using OpenAPI specifications,
executing comprehensive security tests against the target API and reporting vulnerabilities.

OPERATING MODES:

Standalone Mode:
    Use local OpenAPI specification file without Wallarm platform integration.
    Requires: --openapi-file and --target-url flags

    Example:
    docker run -v "$(pwd)/api-spec.yaml:/app/api-spec.yaml" \
        -e WALLARM_API_HOST="https://api.wallarm.com" \
        -e WALLARM_API_TOKEN="..." \
        -e WALLARM_CLIENT_ID="..." \
        wallarm/security-testing:v0.3.2 openapi \
        --openapi-file /app/api-spec.yaml \
        --target-url https://api.example.com

Cloud Mode:
    Integrate with Wallarm platform using Test Policy.
    Requires: WALLARM_TESTING_POLICY_ID environment variable or --testing-policy-id flag

    Example (with local file override):
    docker run -v "$(pwd)/api-spec.yaml:/app/api-spec.yaml" \
        -e WALLARM_API_HOST="..." \
        -e WALLARM_API_TOKEN="..." \
        -e WALLARM_CLIENT_ID="..." \
        -e WALLARM_TESTING_POLICY_ID="..." \
        wallarm/security-testing:v0.3.2 openapi \
        --openapi-file /app/api-spec.yaml

    Example (uses specification from Test Policy):
    docker run \
        -e WALLARM_API_HOST="..." \
        -e WALLARM_API_TOKEN="..." \
        -e WALLARM_CLIENT_ID="..." \
        -e WALLARM_TESTING_POLICY_ID="..." \
        wallarm/security-testing:v0.3.2 openapi

Configuration priority: CLI flags > Environment variables > Test Policy > Defaults

Usage:
security-testing openapi [flags]

Flags:
--auth-header string            Authentication header in 'Name: Value' format (can be provided by Test Policy)
-h, --help                      help for openapi
--openapi-file string           Path to local OpenAPI specification file (JSON or YAML)
--target-url string             Target application URL (can be provided by Test Policy)
--test-types stringSlice        Comma-separated list of test types to run (environment_misconfiguration, sql_injection, command_injection, ssrf, xxe, open_redirect, xss, path_traversal, ssti, lfi_rfi, crlf_injection, infoleak, nosql_injection, rce, graphql_vulnerability_detection)

Global Flags:
API:
--client-id string              Wallarm client ID (required)
--send-logs                     Send logs to Wallarm API (default true)
--testing-policy-id string      Wallarm testing policy ID (optional, enables Test Policy integration)
--wallarm-api-host string       Wallarm API host URL (required)
--wallarm-api-token string      Wallarm API authentication token (required)

GENERAL:
--export-files                  Upload local files (--openapi-file, --postman-collection-file, --postman-environment-files) to Wallarm cloud
--fail-severity string          Set fail severity level (critical, high, medium, low, info) (default "high")

PREFLIGHT-CHECKS:
--skip-target-url-connectivity-check
                                Skip checking target URL connectivity

REPORTS:
--no-summary                    Disable summary output to stdout
-r, --report                    Enable report generation
--report-format string          Report format (json) (default "json")
--report-output string          Report output file path (default "report/results.json")
--report-pretty                 Pretty-print JSON output (default true)

LOGGING:
-f, --log-format string         Set log format (json, colored, text) (default "colored")
-l, --log-level string          Set log level (debug, info, warn, error) (default "info")

Run Postman security testing workflow.

This command performs automated API security testing using Postman collections,
executing comprehensive security tests against the target API and reporting vulnerabilities.

OPERATING MODES:

Standalone Mode:
    Use local Postman collection without Wallarm platform integration.
    Requires: --postman-collection-file flag

    Example:
    docker run -v "$(pwd)/collection.json:/app/collection.json" \
        -e WALLARM_API_HOST="https://api.wallarm.com" \
        -e WALLARM_API_TOKEN="..." \
        -e WALLARM_CLIENT_ID="..." \
        wallarm/security-testing:v0.3.2 postman \
        --postman-collection-file /app/collection.json

Cloud Mode:
    Integrate with Wallarm platform using Test Policy.
    Requires: WALLARM_TESTING_POLICY_ID environment variable or --testing-policy-id flag

    Example (with local files override):
    docker run -v "$(pwd)/collection.json:/app/collection.json" \
        -v "$(pwd)/env.json:/app/env.json" \
        -e WALLARM_API_HOST="..." \
        -e WALLARM_API_TOKEN="..." \
        -e WALLARM_CLIENT_ID="..." \
        -e WALLARM_TESTING_POLICY_ID="..." \
        wallarm/security-testing:v0.3.2 postman \
        --postman-collection-file /app/collection.json \
        --postman-environment-files /app/env.json

    Example (uses collection from Test Policy):
    docker run \
        -e WALLARM_API_HOST="..." \
        -e WALLARM_API_TOKEN="..." \
        -e WALLARM_CLIENT_ID="..." \
        -e WALLARM_TESTING_POLICY_ID="..." \
        wallarm/security-testing:v0.3.2 postman

Configuration priority: CLI flags > Environment variables > Test Policy > Defaults

Usage:
security-testing postman [flags]

Flags:
-h, --help                      help for postman
--postman-collection-file string  Path to local Postman collection file (JSON)
--postman-environment-files stringSlice
                                Path(s) to Postman environment file(s) (JSON, supports multiple files)



Global Flags:
API:
--client-id string              Wallarm client ID (required)
--send-logs                     Send logs to Wallarm API (default true)
--testing-policy-id string      Wallarm testing policy ID (optional, enables Test Policy integration)
--wallarm-api-host string       Wallarm API host URL (required)
--wallarm-api-token string      Wallarm API authentication token (required)

GENERAL:
--export-files                  Upload local files (--openapi-file, --postman-collection-file, --postman-environment-files) to Wallarm cloud
--fail-severity string          Set fail severity level (critical, high, medium, low, info) (default "high")

PREFLIGHT-CHECKS:
--skip-target-url-connectivity-check
                                Skip checking target URL connectivity

REPORTS:
--no-summary                    Disable summary output to stdout
-r, --report                    Enable report generation
--report-format string          Report format (json) (default "json")
--report-output string          Report output file path (default "report/results.json")
--report-pretty                 Pretty-print JSON output (default true)

LOGGING:
-f, --log-format string         Set log format (json, colored, text) (default "colored")
-l, --log-level string          Set log level (debug, info, warn, error) (default "info")

See the explanation for the -fail-severity flag in the next section.

Test run success criteria¶

If your include Wallarm's Schema-Based test runs into your CI/CD pipeline, it is important to define the criteria of test run success. You can do that in the test policy details by setting
the Fail on a risk level to one of the values:

• critical

• high (default)

• medium

• low

• info

Selecting any value different from default adds the FAIL_SEVERITY environment variable to the Docker run command.

Example:

docker run -e WALLARM_API_HOST="us1.api.wallarm.com" -e WALLARM_API_TOKEN="<YOUR_SCHEMA-BASED_TESTING_AGENT-TOKEN>" -e WALLARM_CLIENT_ID="<YOUR_COMPANY_ID>" -e WALLARM_TESTING_POLICY_ID="<YOUR_TEST_POLICY_ID>" -e FAIL_SEVERITY="medium" --network="host" wallarm/security-testing:0.3.2

You can also set test run success criteria with the -fail-severity flag, for example:

docker run -e WALLARM_API_HOST="us1.api.wallarm.com" -e WALLARM_API_TOKEN="<YOUR_SCHEMA-BASED_TESTING_AGENT-TOKEN>" -e WALLARM_CLIENT_ID="<YOUR_COMPANY_ID>" -e WALLARM_TESTING_POLICY_ID="<YOUR_TEST_POLICY_ID>" --network="host" wallarm/security-testing:0.3.2 -fail-severity medium

...will lead to the following: if testing finds at least 1 medium (or high or critical) security issue, the test run will exit with code 1 ("test failed").

Deleting policies¶

You can delete a test policy. If you do so:

• Information on previous test runs will remain untouched

• You will not be able to run Docker's command based on the deleted policy

• If policy's Docker containers are running, they will continue to do so and the testing will continue

• When policy's Docker containers stop, you will not be able to re-run them

Was this page helpful?

How can we improve this article? (Optional)

Information is unclear or confusing

Insufficient information

Outdated or incorrect information

0/240

Send