README.md

# `ci/component/blackduck`

> A GitLab CI component for running BlackDuck scans

## Getting Started

Add the CI component to the `.gitlab-ci.yml`:

```yaml
include:
  - component: "gitlab.arm.com/ci/component/blackduck@<version>"
```

Add a job that extends the `.blackduck` component to perform the scan:

```yaml
blackduck_scan:
  extends: .blackduck
```

## Variables

The template has some environment variables that can customise the job behaviour.

### `VERSION_NAME`

To run `blackduck` scan with multiple variable names:

```yaml
blackduck_scan:
  extends: .blackduck
  parallel:
    matrix:
      - VERSION_NAME:
          - v1.0.0
          - v1.1.0
```

## Purpose

Component job to run blackduck detect scan.

### Introduction

Blackduck Hub is a Synopsys Integrated platform for providing code snippet matching, security scanning and SCA binary
analysis.

It helps with managing the supply chain of software, understanding the third-party components in use and minimizing
risks from known vulnerabilities and licensing.  
Black Duck is a comprehensive solution for supply chain management, based primarily on source analysis.

Using Black Duck, you can:

- Scan your code and identify open source software that exists in your code base.
- View the generated Bill of Materials (BOM) for your software projects.
- View vulnerabilities that have been identified in open source components.
- Assess your security, license, and operational risk. It integrates with the preferred language dependency management
  system (`golang`, `maven`, `docker` etc) and will automatically detect standard blobs of TPIP brought in this way to
  avoid having to manually declare them

There is also TPIP CODE SNIPPET MATCHING as an extension which is comparable to what Protex did. Identified snippets
lead to a component being listed on the related BOM, and vulnerabilities flagged.

Black Duck will also carry out license identification based on components (using headline license match, no deeper
information), but does not currently have Protex’s deep license compliance functionality

### Detect script

To scan code Blackduck uses its detect script.

Detect is Black Duck's intelligent scan client that scans code bases in your projects and folders to perform
compositional analysis.

Synopsys Detect sends scan results to Black Duck, which generates risk analysis when identifying open source components,
licenses, and security vulnerabilities.

For more info see: [blackduck][blackduck]

### Detector Notes

Blackduck's detect script uses detectors to find and extract dependencies from all supported package managers.

Each package manager ecosystem is assigned a detector type. Each detector type may have multiple methods used to extract
dependencies that have their own requirements themselves.

Make sure to review the requirements of the code you're interested to scan.

Detectors are split into detectors that need package managers installing and detectors that don't: e.g. NPM is buildless
and doesn't need anything installing.

For more info see: [detectors][detectors]

#### Detector Gotchas

If the detector needs packages installing and you haven't installed those packages (e.g. a CI job just doing a git
clone) then the detector won't scan these packages.

##### Pip detector

The Pip Detector needs all the Python packages installed by pip so they show up in pip list.

If you don't install the components needed as part of the scan they won't get scanned,

e.g. a `requirements.txt` with no pip installs wouldn't scan the dependencies.

##### Poetry detector

If the detector discovers a `pyproject.toml` file but not a `poetry.lock` file, it will prompt the user to generate a
`poetry.lock` by running `poetry install` and then run Detect again.

##### NPM detector

If the detector discovers a `package.json` file but not the relevant `node_modules` folder the user will get a scan with
low accuracy. To fix this install the `node_modules` (normally only need production `node_modules`) and run again.

## Usage

### Configuration

#### Blackduck Detect Variables

Review [detect documentation] for more details

##### Detect Blackduck server variables

The detect script needs to be configured to reach the blackduck server and project by setting variables `BLACKDUCK_URL`,
`BLACKDUCK_API_TOKEN`, `BLACKDUCK_PROJECT_NAME`

#### `BLACKDUCK_URL`

The URL of the blackduck server.

This variable (e.g. `https://blackduck-server.com`) should be set as a secret in Gitlab's CI secrets.

Will only need to be updated if blackduck's URL is updated

#### `BLACKDUCK_API_TOKEN`

The access token used to authenticate with the Blackduck Server.

This variable (e.g. `BLACKDUCK_API_TOKEN`) should be set as a secret in Gitlab's CI secrets and marked as protected and
masked.

#### `BLACKDUCK_PROJECT_NAME`

The Blackduck project name to be used in the Blackduck server.

##### Detect project scan settings

Various options can be configured within the detect scan itself

#### `VERSION_NAME`

Name of the scan itself in blackduck, default is `CI_COMMIT_REF_NAME` (the branch name).

#### `PATH_TO_SCAN`

The relative path to directories to scan, default is the project root dir.

#### `PATH_TO_EXCLUDE`

The relative path to any directories that you don't want to scan, default is to leave empty.

#### `UPLOAD_SOURCE`

Upload the source code to the blackduck server, default is `false`. Options are: `true`, `false`.

#### `SNIPPET_MATCHING`

Snippet matching finds fragments of open source code used in your proprietary files (or open source files moved into
your proprietary directories) and matches that code with open source code found in one or more Black Duck KnowledgeBase
files.

n.b. While the depth of the detectors is set in the CLI, the depth of the Snippet matching is set in the UI in settings.

Default is `FULL_SNIPPET_MATCHING`. Options are: `NONE`, `SNIPPET_MATCHING`, `SNIPPET_MATCHING_ONLY`
`FULL_SNIPPET_MATCHING` `FULL_SNIPPET_MATCHING_ONLY`.

#### `SEARCH_DEPTH`

Depth of subdirectories within the source directory to which Detect will search for files.

Default is `20` (n.b. Blackduck default is 0 meaning no subdirectories would be scanned by detectors)

#### `LOGGING_MODE`

Logging mode. Useful for debugging errors.

Default is `INFO`. Options are: `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`, `FATAL`, `OFF`.

#### `SCAN_MODE`

It's possible to run the Blackduck detect script in either `INTELLIGENT` or `RAPID` mode.

`INTELLIGENT` scan is an asynchronous scan where the results are uploaded to the Blackduck client for analysis at a
later time.

`RAPID` scan [rapid scan][rapid scan] is a synchronous scan that produces results of any security or license
vulnerabilities at the time of running the script, when running the rapid scan a team might want to override the default
rules of when to run.

Default is `INTELLIGENT`. Options are: `RAPID`, `INTELLIGENT`.

##### Rapid Scan Gotchas

###### Policies

Policies allow you to decide what open source is acceptable for use.

This can be helpful for managing security, license, and operational risk, by enabling you to create rules to govern the
use of open source components in your projects.

https://community.synopsys.com/s/article/Black-Duck-Configuring-Policy-Management

For rapid scans to work a policy must be set on the project, e.g. a policy of:

- License Risk Greater Than Medium
- Medium or Higher Severity Vulnerability Count Greater than 0

see: [policy management][policy management]

### Scheduled Pipelines

This job can be run in a scheduled pipelines (useful for `INTELLIGENT` scans)

When this pipeline is created it should be created with variable: `CI_SCHEDULE_NAME` which should be set to
`blackduck-scan` or can be run on every commit (more useful with `RAPID` scans)

see: [Scheduled pipelines docs][scheduled pipelines]

### Example

Example to upload a scan of a Python + Node project with a `src` directory to a Blackduck project called `test-project`.
The example is a Python + Node project so we do a `pip install` and `npm install` before the scan. `BLACKDUCK_API_TOKEN`
and `BLACKDUCK_URL` have been set in gitlab ci vars and creating the scheduled pipeline with variable `CI_SCHEDULE_NAME`
set to `blackduck-scan`

```yaml
include:
  - component: "gitlab.arm.com/ci/component/blackduck@<version>"
    inputs:
      BLACKDUCK_PROJECT_NAME: "test-project"
      PATH_TO_SCAN: "./src"
      PATH_TO_EXCLUDE: "./src/tests/"
      SCAN_MODE: "RAPID"

.blackduck-example-project:
  extends:
    - .blackduck
  before_script:
    - python3 -m venv venv
    - source venv/bin/activate
    - pip list
    - pip install -r ${ROOT}/requirements.txt
    - pip list
    - npm i --production
    - npm list

blackduck-rapid-scan:
  extends:
    - .blackduck-example-project
  rules:
    - if: $CI_MERGE_REQUEST_IID
      changes:
        - requirements.txt
        - package.json

blackduck-intelligent-scan:
  extends:
    - .blackduck-example-project
```


When scheduled Blackduck would do an `INTELLIGENT` scan on the directory `src` and publish the results to
`test-project`. If `requirements.txt` or `package.json` were changed Blackduck would do an `RAPID` scan on the Python +
Node packages as part of the MR.

[blackduck]: https://github.com/blackducksoftware/synopsys-detect
[detect documentation]:
  https://community.synopsys.com/s/document-item?bundleId=integrations-detect&topicId=introduction.html&_LANG=enus
[scheduled pipelines]: https://docs.gitlab.com/ee/ci/pipelines/schedules.html
[detectors]: https://sig-product-docs.synopsys.com/bundle/integrations-detect/page/components/detectors.html
[rapid scan]: https://sig-product-docs.synopsys.com/bundle/integrations-detect/page/runningdetect/rapidscan.html
[policy management]: https://community.synopsys.com/s/article/Black-Duck-Configuring-Policy-Management