Repository navigation
⚙️ Add a generic (SCB) finding importer to the DefectDojo Integration Hook #332
Description
Is your feature request related to a problem?
As secureCodeBox (SCB) user i would like to use all integrated security scanners and examine their results in OWASP DefectDojo (DD), when i use DD as vulnerability management tool in my environment.
With the latest PR #300 introducing the DefectDojo-Persistence Hook it is now already possible to import some SCB scanner findings supported by DefectDojo:
Problem is that there are some scanners missing which are already integrated within secureCodeBox but have no corresponding parser at OWASP DefectDojo. Thats why i'm currently not able to import and analyse the following scanner finding results. Using those scanners in combination with the DefectDojo-Persistence Hook leads to failed scans:
- WPScan
- SSH-Scan
- Nikto (in JSON format instead of XML)
- Kube-Hunter
- Kubeaudit
Example failure
Example scan with kube-hunter:
k tree scan kube-hunter-internal-1616236981 -n demo-scans
NAMESPACE NAME READY REASON AGE
demo-scans Scan/kube-hunter-internal-1616236981 - 96m
demo-scans ├─Job/defectdojo-hook-kube-hunter-internal-1616236981-hwnhw - 95m
demo-scans │ ├─Pod/defectdojo-hook-kube-hunter-internal-1616236981-hwnhw-8cthv False ContainersNotReady 95m
demo-scans │ ├─Pod/defectdojo-hook-kube-hunter-internal-1616236981-hwnhw-gjdz6 False ContainersNotReady 95m
demo-scans │ ├─Pod/defectdojo-hook-kube-hunter-internal-1616236981-hwnhw-hf26w False ContainersNotReady 93m
demo-scans │ └─Pod/defectdojo-hook-kube-hunter-internal-1616236981-hwnhw-xdszp False ContainersNotReady 94m
demo-scans └─Job/parse-kube-hunter-internal-1616236981-qnwc8 - 95m
demo-scans └─Pod/parse-kube-hunter-internal-1616236981-qnwc8-wm4ls False PodCompleted 95mExample defectdojo-hook log:
2021-03-20 10:45:42 DEBUG RestTemplate:147 - Accept=[text/plain, application/json, application/*+json, */*]
2021-03-20 10:45:42 DEBUG RestTemplate:147 - Response 200 OK
2021-03-20 10:45:42 DEBUG RestTemplate:147 - Reading to [java.lang.String] as "application/octet-stream"
2021-03-20 10:45:42 DEBUG VersionedEngagementsStrategy:99 - Finished Downloading Scan Report (RawResults)
Exception in thread "main" java.lang.IllegalArgumentException: No Mapping found for ScanType 'kube-hunter'
at io.securecodebox.persistence.util.ScanNameMapping.bySecureCodeBoxScanType(ScanNameMapping.java:60)
at io.securecodebox.persistence.strategies.VersionedEngagementsStrategy.createTest(VersionedEngagementsStrategy.java:259)
at io.securecodebox.persistence.strategies.VersionedEngagementsStrategy.run(VersionedEngagementsStrategy.java:101)
at io.securecodebox.persistence.DefectDojoPersistenceProvider.main(DefectDojoPersistenceProvider.java:53)Describe alternatives you've considered
The following alternative solutions are only focussed on the missing parser problem:
There are multiple solution strategies to solve this problem:
- Use the generic CSV findings importer to implement a generic SCB findings import in the DefectDojo-Persistence Hook
public enum ScanNameMapping {
NMAP("nmap", ScanType.NMAP_SCAN),
ZAP_BASELINE("zap-baseline", ScanType.ZAP_SCAN),
ZAP_API_SCAN("zap-api-scan", ScanType.ZAP_SCAN),
ZAP_FULL_SCAN("zap-full-scan", ScanType.ZAP_SCAN),
SSLYZE("sslyze", ScanType.SS_LYZE_3_SCAN_JSON),
TRIVY("trivy", ScanType.TRIVY_SCAN),
GITLEAKS("gitleaks", ScanType.GITLEAKS_SCAN),
// New Approach
// NIKTO("nikto", ScanType.GENERIC_CSV_SCAN),
// SSH("ssh-scan, ScanType.GENERIC_CSV_SCAN),
;- You will find the existing generic CSV DefectDojo parser here: https://github.com/DefectDojo/django-DefectDojo/tree/master/dojo/tools/generic
- Implement a new generic JSON Findings Importer in OWASP DefectDojo as already suggested here Add Generic JSON importer DefectDojo/django-DefectDojo#3798 and use it to implement a generic SCB findings import in the DefectDojo-Persistence Hook
public enum ScanNameMapping {
NMAP("nmap", ScanType.NMAP_SCAN),
ZAP_BASELINE("zap-baseline", ScanType.ZAP_SCAN),
ZAP_API_SCAN("zap-api-scan", ScanType.ZAP_SCAN),
ZAP_FULL_SCAN("zap-full-scan", ScanType.ZAP_SCAN),
SSLYZE("sslyze", ScanType.SS_LYZE_3_SCAN_JSON),
TRIVY("trivy", ScanType.TRIVY_SCAN),
GITLEAKS("gitleaks", ScanType.GITLEAKS_SCAN),
// New Approach
// NIKTO("nikto", ScanType.GENERIC_JSON_SCAN),
// SSH("ssh-scan, ScanType.GENERIC_JSON_SCAN),
;- Such a new generic JSON DefectDojo parser should be implemented here: https://github.com/DefectDojo/django-DefectDojo/tree/master/dojo/tools/
- Implement a new generic SCB JSON Findings Importer in OWASP DefectDojo which is based on the secureCodeBox specific findings format and use it to import them in the DefectDojo-Persistence Hook
public enum ScanNameMapping {
NMAP("nmap", ScanType.NMAP_SCAN),
ZAP_BASELINE("zap-baseline", ScanType.ZAP_SCAN),
ZAP_API_SCAN("zap-api-scan", ScanType.ZAP_SCAN),
ZAP_FULL_SCAN("zap-full-scan", ScanType.ZAP_SCAN),
SSLYZE("sslyze", ScanType.SS_LYZE_3_SCAN_JSON),
TRIVY("trivy", ScanType.TRIVY_SCAN),
GITLEAKS("gitleaks", ScanType.GITLEAKS_SCAN),
// New Approach
// NIKTO("nikto", ScanType.GENERIC_SCB_SCAN),
// SSH("ssh-scan, ScanType.GENERIC_SCB_SCAN),
;- Such a new generic SCB DefectDojo parser should be implemented here: https://github.com/DefectDojo/django-DefectDojo/tree/master/dojo/tools/
Describe the solution you'd like
I would prefer the solution alternative 3 because it seems to be a clean integration strategy. Both OWASP projects (secureCodeBox and DefectDojo) would have advantages.
Additional context
- Documentation of all currently supported security scanner parsers implemented by OWASP DefectDojo: https://defectdojo.readthedocs.io/en/latest/integrations.html
- secureCodeBox Java Client for DefectDojo: https://github.com/secureCodeBox/defectdojo-client-java
Activity
madchap commentedon May 1, 2021
Nikto should work with JSON: https://github.com/DefectDojo/django-DefectDojo/blob/master/dojo/tools/nikto/parser.py#L42
w.r.t to integration, it seems alternative 3 would indeed offer the greatest amount of flexibility and maintainability for SCB.
rfelber commentedon May 1, 2021
Hi @madchap ,
thx for your hint 👍🏻
@JohannesZahn is already working on alternative 3 🎉 🤗
https://github.com/secureCodeBox/django-DefectDojo/blob/parser-SCB/dojo/tools/securecodebox/parser.py
A corresponding DefectDojo PR will follow in the next week if everything works out as expected 🤞
damiencarol commentedon May 3, 2021
Hello gents, thanks for all the details and the comments.
Interested to see what kind of parser it will make.
I was thinking that it could be better to rely on the generic parser for both projects in the long term as it exists
since a long time and is tested on our side. I'm wondering what were missing in the generic one that were you consider writing one from scratch @JohannesZahn
As @madchap said, implementing a parser from scratch will give you flexibility but with a maintenance cost.
For the JSON format of Nikto, as @madchap mentionned it's working and tested in dojo. It rely on the file name.
It should work as-is. (Available to test if/why not :) )
On a more higher level, @rseedorff do you plan to have some kind of test/demo env with both products?
JohannesZahn commentedon May 7, 2021
Good morning @damiencarol
to use the generic parser we would have to export our findings in your generic csv format if I'm not mistaken. The json file we import into DefectDojo with the new parser already existed and will be maintained anyway. So exporting our findings in csv would also introduce maintanance cost. As you already provide tools (cookiecutter) for integrating new parsers and because of the flexibility @rseedorff and me decided to go with that option.
I will also integrate the Nikto JSON Parser thanks for the hint with the filename 👍
damiencarol commentedon May 7, 2021
2 questions:
Also regarding the documentation, I didn't found more details about the finding format of SCB only this one page https://docs.securecodebox.io/docs/api/finding/ that is just an example and don't describe attributes.
Also, if we chose this path, there is some attributes that do not exists in the SCB format but are important for DefectDojo like the date of the finding (
date). Should we make proposals to add more data in SCB format?rfelber commentedon May 9, 2021
Hi @damiencarol,
thx for your feedback and hints! We appreciate working so closely together :)
Our secureCodeBox findings format is stable (no changes) and flexible enough to be extended on demand for new scanners.
We are searching for even more "DAST FINDING FORMAT" standardisation (#322) but as far as I know this there is only a SAST standard available (SARIF) 🤔.
Adding a detailed format description and a json schema for validation is a great idea, I will add this as new issue to our backlog 👍🏻
If you have any suggestions about missing field we would've happy about any proposals. The
date/timestampof the finding is already part of the format, but not documented as it seems 🤨Regarding you question about an demo env. Yes, for sure 😊 We already have an non public cloud demo environment (SCB + DefectDojo) running and want to create a public one too. Does DefectDojo have something similar? Maybe we can think about combining both environments 🤔
damiencarol commentedon May 9, 2021
Hi!
We're also glad to have some synergy with SCB project (it's a shared feeling in the core team). I push to have our env with SCB running live. It's https://github.com/mtesauro that manage our demo env. I shared this info with the core team that we want to build something in common. Maybe we could also share the maintenance of this demo env.
Standards
Regarding the standard available, I share your feeling. I didn't found a format that is the equivalent of SARIF for DAST.
👏👏👏
Happy to see the
date. Could be good to push @JohannesZahn to add implementation+unit tests for this attribute in his PR.Another big missing part is the Request/Answer part.
Let me give more details; some parsers give the details of how they detected a vulnerabilities. Like for ex the parameters used, URL, etc... We have specific data structures to support these data. Our UI manage well and show these requests/answers associated with the finding.
The big question is: does your format support that?
JohannesZahn commentedon May 18, 2021
I will implement the Date in the Parser and get back to you for advice on improving the SCB Findings Format @damiencarol.
Also before we submit our Parser PR we would like to evaluate the new DD generic json parser and see if it works well for us.
damiencarol commentedon Jun 5, 2021
@JohannesZahn we already merged the generic JSON parser if you want to test it (only i
devfor now, it will be part of2.0.0release at the end of the month)JohannesZahn commentedon Jun 8, 2021
@damiencarol thats awesome, thank you. I am currently integrating it into SCB and so far everything looks good.