1. Overview
The purpose of Hash Sharing is to provide a system by which image and video fingerprints can be collected, maintained, and distributed among organizations.
1.2. Changes Since v1.0
Version 2.0 of Hash Sharing offers the ability to share video file fingerprints.
To provide more flexibility for evolving features, a new schema and set of endpoints were introduced.
Industry and NPO environment schemas were consolidated into a single schema with namespace https://hashsharing.ncmec.org/hashsharing/v2, and the new endpoints were made available at /v2/.
Version 2.0 introduces the ability to control access to submitted entries by other members.
Version 2.0 introduces pagination of query results.
Version 2.0 introduces the ability to annotate entries with categorizations.
Version 2.0 introduces the ability to retract many entries at once in the same request.
Version 2.0 introduces the ability to submit and retrieve larger fingerprints.
Version 2.0 introduces the ability to submit and view feedback on other’s entries.
Submitted image entries are available to both v1.0 and v2.0 users; entries submitted using v1.0 are available to v2.0 users, and those submitted to v2.0 are available to v1.0 users. Users who have submitted images using v1.0 may also update or retract entries using v2.0. As v1.0 schemas and endpoints do not support videos, video entries are only available to v2.0 endpoint users.
2. Access
For production, the application base URL is one of the following.
| Production Environments | |
|---|---|
Industry CSAM |
|
Industry Exploitative |
|
Law Enforcement CSAM |
|
NPO CSAM |
|
NPO Exploitative |
|
For testing, the application base URL is one of the following.
| Test Environments | |
|---|---|
Industry CSAM |
|
Industry Exploitative |
|
Law Enforcement CSAM |
|
NPO CSAM |
|
NPO Exploitative |
|
This documentation uses https://{applicationRoot} as a representative base URL, which should be replaced with the appropriate production or testing environment URL from above.
Endpoints are only accessible via HTTPS.
Authentication is required and a username and password must be requested from and supplied by NCMEC. Members are also granted either read-only or read/write access separately within each environment.
| Credentials appearing in this document are for the purpose of example only. You will not be able to use them to authenticate in either the testing or production environments. |
3. Terminology
The following definitions are provided for clarification of terms used in this document.
- Entry
-
The combination of an entry ID, one or more file fingerprints, and an optional classification of the file contents. Within XML received and sent by Hash Sharing, an entry is the element
<image>or<video>and its contents. - Entry ID
-
The ID for an entry specified by the submitting member. The submitting member and entry ID form the unique key for an entry within Hash Sharing. Within XML received and sent by Hash Sharing, an entry ID is the element
<id>within the element<image>or<video>. Entry IDs are specific to the submitting member. An entry ID for an image submitted by a member cannot also be used by a video submitted by that same member and vice versa. - Submission
-
A list of entries sent to Hash Sharing by a member for submission into the database.
4. Endpoints
The Hash Sharing API is RESTful and provides the endpoints listed below relative to one of the base URLs indicated above.
For example, to make a submission of both image and video entries, construct a POST request to the URL https://{applicationRoot}/v2/entries.
More detailed examples are provided in a later section.
4.1. General
- GET
/v2/schema.xsd -
Download the latest XML schema definition.
- GET
/v2/status -
Return information about the current user.
- GET
/v2/members -
Return the list of members.
4.2. Entries
- GET
/v2/entries?{parameters} -
Return a list of entries that match the specified query parameters.
| Parameter | Description |
|---|---|
|
- GET
/v2/entries/images?{parameters} -
Return a list of image entries that match the specified query parameters.
| Parameter | Description |
|---|---|
|
- GET
/v2/entries/videos?{parameters} -
Return a list of video entries that match the specified query parameters.
| Parameter | Description |
|---|---|
|
- POST
/v2/entries -
Submit entries. The submission is a single XML document conforming to the web service schema. A response from the server will indicate the count of entries that were received, accepted, inserted, and updated. If any entries were rejected, the response will indicate which entries were not accepted and why each was not accepted.
- POST
/v2/entries/images -
Submit image entries. Entry elements within the submitted XML document that are not image entries are ignored.
- POST
/v2/entries/videos -
Submit video entries. Entry elements within the submitted XML document that are not video entries are ignored.
- DELETE
/v2/entries/{entryId} -
Retract an entry you have submitted. If an entry is not found for the current user or is already retracted, this operation does nothing.
| Parameter | Description |
|---|---|
|
the Entry ID you specified when submitting the entry |
- DELETE
/v2/entries/images/{entryId} -
Retract an image entry you have submitted. If an entry is not found for the current user, is already retracted, or is not an image, this operation does nothing.
| Parameter | Description |
|---|---|
|
the Entry ID you specified when submitting the entry |
- DELETE
/v2/entries/images/{entryId} -
Retract a video entry you have submitted. If an entry is not found for the current user, is already retracted, or is not a video, this operation does nothing.
| Parameter | Description |
|---|---|
|
the Entry ID you specified when submitting the entry |
- DELETE
/v2/entries -
Retract entries you have submitted and whose entry IDs match those present in the submitted XML document. If an entry is not found for the current user, is already retracted, or is not present, it will be skipped. The remaining valid entries will be retracted.
- DELETE
/v2/entries/images -
Retract image entries you have submitted and whose entry IDs match those present in the submitted XML document. If an entry is not found for the current user, is already retracted, or is not an image, it will be skipped. The remaining valid entries will be retracted.
- DELETE
/v2/entries/videos -
Retract video entries you have submitted and whose entry IDs match those present in the submitted XML document. If an entry is not found for the current user, is already retracted, or is not a video, it will be skipped. The remaining valid entries will be retracted.
4.3. Fingerprints
- GET
{fingerprintPath} -
Retrieve the raw binary data for a large fingerprint. See the section on working with large fingerprints for more details.
| Parameter | Description |
|---|---|
|
the path, relative to the application root URL, to the fingerprint provided in an entry query response |
- PATCH
/v2/entries/{entryId}/fingerprints/{fingerprintType} -
Add a fingerprint to an entry you have submitted. The request body is the raw binary data for the fingerprint. Multipart requests are not supported.
| Parameter | Description |
|---|---|
|
the Entry ID you specified when submitting the entry |
|
the type of the fingerprint |
- DELETE
/v2/entries/{memberEntryId}/fingerprints/{fingerprintType} -
Remove a fingerprint from an entry you have submitted.
| Parameter | Description |
|---|---|
|
the Entry ID you specified when submitting the entry |
|
the type of the fingerprint |
4.4. Categorizations
- GET
/v2/categorizations -
Returns the list of all available categorizations that may be used to annotate entries. This includes their groups and any restrictions between categorizations.
- PUT
/v2/entries/{memberEntryId}/categorizations -
Assigns categorizations to an entry you have already submitted. Categorizations must be one of the ones supplied by the endpoint above.
| Parameter | Description |
|---|---|
|
the member Entry ID you specified when submitting the entry |
4.5. Feedback
- GET
/v2/feedback/{feedbackType}/reasons -
Return the list of available downvote reasons.
| Parameter | Description |
|---|---|
|
the name of the feedback type |
- PUT
/v2/entries/{memberId}/{memberEntryId}/{feedbackType}/feedback -
Allows a member to provide feedback for an entry using a specific feedback type.
| Parameter | Description |
|---|---|
|
the ID of the member for whom you are providing feedback to |
|
the member’s chosen Entry ID specified when they submitted the entry |
|
the name of the feedback type |
4.6. Access Permissions
- GET
/v2/permissions/access -
Return the list of members that you have explicitly granted access to your entries. If you have provided no explicit grants, the response will be empty, and all other members will be able to access your entries. To remove the ability for all other members to access your entries, grant only yourself access to your entries using your own member ID.
- POST
/v2/permissions/access/{memberId} -
Grant access to your entries to a member. If you provide no explicit grants, all other members will be able to access your entries. To remove the ability for all other members to access your entries, grant only yourself access to your entries using your own member ID.
| Parameter | Description |
|---|---|
|
the ID of the member to whom you are granting access |
- DELETE
/v2/permissions/access/{memberId} -
Revoke previously granted access to your entries from a member. If you remove all explicit grants, all other members will be able to access your entries. To remove the ability for all other members to access your entries, grant only yourself access to your entries using your own member ID.
| Parameter | Description |
|---|---|
|
the ID of the member from whom you are revoking access |
5. Supported Query Parameters
Supported query parameters are listed below.
Parameter keys are case-sensitive.
Supported query parameters apply to all entry query endpoints: /v2/entries, /v2/entries/images, and /v2/entries/videos.
fromrequired-
Selects entries that were created or updated after this date and time, inclusive. The value of this parameter must be in ISO 8601 format
yyyy-MM-dd’T’HH:mm:ss.SSSZ. This parameter is required. torequired-
Selects entries that were created or updated before this date and time, exclusive. The value of this parameter must be in ISO 8601 format
yyyy-MM-dd’T’HH:mm:ss.SSSZ. This parameter is required. memberId-
Selects entries that were created by the specified member. A list of members and their IDs may be obtained via the
/v2/membersendpoint. This parameter is optional.
6. Available Data Formats
Server responses can be requested as either XML or JSON.
To indicate the desired response for server responses, specify an Accept header.
To request JSON responses, specify an Accept header with value application/json.
To explicitly request XML responses, specify an Accept header with value application/xml.
If no Accept header is specified, the default response type is XML.
| The primarily supported data format is XML. XML is the format used throughout this document and is the only supported data format for request body payloads other than fingerprint uploads. |
7. Examples
To illustrate how Hash Sharing is used, included are examples using curl. Executables and documentation for curl are available at https://curl.haxx.se. These examples will show how to query entries, submit entries, update entries, and retract entries. For this illustration, the following endpoint and example credentials will be used. Note that the username and password below are not valid credentials; you can use your NCMEC-provided credentials to follow along with this example.
-
The example base URL is the test environment: https://hashsharing-test.ncmec.org/npo
-
The example username and password supplied by NCMEC is
usr123andpswd123
| The credentials used in this section are for illustration only. You will not be able to use them to authenticate in either the testing or production environments. |
| In these examples, some fingerprints have been truncated for readability. Also note that the data provided in this document is used for illustration only; you can expect to retrieve different data when making the same requests during testing. |
7.1. Querying Entries
Hash Sharing supports finding entries by making a query for a date range in which entries were submitted or last updated, regardless of the member that submitted them.
To query entries, make a GET request to /v2/entries?{parameters}.
The from and to parameters are required and must both in the ISO 8601 format yyyy-MM-dd’T’HH:mm:ss.SSSZ and URL-encoded.
See Supported Query Parameters for a list of all supported {parameters} options.
In this example, we query for entries of any type submitted or last updated between 2017-10-20 00:00:00Z and 2017-10-30 00:00:00Z.
The curl call to execute this query is shown below.
curl --user usr123:pswd123 \
--data-urlencode "from=2017-10-20T00:00:00.000Z" \
--data-urlencode "to=2017-10-30T00:00:00.000Z" \
--get https://hashsharing-test.ncmec.org/npo/v2/entriesThe response includes a list of active entries and list of entries that were retracted during the specified period, grouped by their entry type.
Each entry type group also includes the attributes count (the count of all entries for that type on this page of results, whether active or retracted) and maxTimestamp (the most recent timestamp of the query results for that type on this page).
The maximum maxTimestamp from all pages of results may be used as the from parameter in the user’s next query to prevent gaps between queries.
<?xml version="1.0" encoding="UTF-8"?>
<queryResult xmlns="https://hashsharing.ncmec.org/hashsharing/v2">
<images count="4" maxTimestamp="2017-10-24T15:10:00Z">
<image>
<member id="42">Example Member</member>
<timestamp>2017-10-24T15:00:00Z</timestamp>
<id>image1</id>
<classification>A1</classification>
<fingerprints>
<md5>a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1</md5>
<sha1>a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1</sha1>
<pdna>a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1...</pdna>
</fingerprints>
<categorizations>
<categorization guid="01234567-abcd-0123-4567-0123456789ab" name="Example Categorization 1"/>
<categorization guid="01255555-abcd-0123-4567-0123456789cd" name="Example Categorization 2"/>
</categorizations>
<feedback lastUpdateTimestamp="2023-06-01T12:48:14Z">
<affirmativeFeedback type="Md5" lastUpdateTimestamp="2023-06-01T12:48:14Z">
<members timestamp="2023-06-01T12:48:14Z">
<member id="42">Example Member</member>
</members>
</affirmativeFeedback>
<negativeFeedback type="Sha1" lastUpdateTimestamp="2023-05-31T19:41:51Z">
<reasons>
<reason guid="01234567-abcd-0123-4567-012345678900" name="Example Reason 1" type="Sha1"/>
<members timestamp="2023-06-01T12:48:14Z">
<member id="42">Example Member</member>
</members>
</reasons>
</negativeFeedback>
</feedback>
</image>
<image>
<member id="42">Example Member</member>
<timestamp>2017-10-24T15:00:00Z</timestamp>
<id>image2</id>
<classification>A2</classification>
<fingerprints>
<md5>a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2</md5>
<sha1>a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2</sha1>
<pdna>a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2...</pdna>
</fingerprints>
<categorizations />
<feedback />
</image>
<image>
<member id="42">Example Member</member>
<timestamp>2017-10-24T15:00:00Z</timestamp>
<id>image3</id>
<classification>B1</classification>
<fingerprints>
<md5>a3a3a3a3a3a3a3a3a3a3a3a3a3a3a3a3</md5>
<sha1>a3a3a3a3a3a3a3a3a3a3a3a3a3a3a3a3a3a3a3a3</sha1>
<pdna>a3a3a3a3a3a3a3a3a3a3a3a3a3a3a3a3a3a3a3a3a3a3a3a3a3a3a3...</pdna>
</fingerprints>
<categorizations />
<feedback />
</image>
<deletedImage>
<member id="42">Example Member</member>
<id>image4</id>
<timestamp>2017-10-24T15:10:00Z</timestamp>
</deletedImage>
</images>
<videos count="4" maxTimestamp="2017-10-24T15:20:00Z">
<video>
<member id="42">Example Member</member>
<timestamp>2017-10-24T15:00:00Z</timestamp>
<id>video1</id>
<fingerprints>
<md5>b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1</md5>
<sha1>b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1</sha1>
</fingerprints>
</video>
<video>
<member id="42">Example Member</member>
<timestamp>2017-10-24T15:00:00Z</timestamp>
<id>video2</id>
<fingerprints>
<md5>b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2</md5>
<sha1>b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2</sha1>
</fingerprints>
</video>
<video>
<member id="42">Example Member</member>
<timestamp>2017-10-24T15:00:00Z</timestamp>
<id>video3</id>
<fingerprints>
<md5>b3b3b3b3b3b3b3b3b3b3b3b3b3b3b3b3</md5>
<sha1>b3b3b3b3b3b3b3b3b3b3b3b3b3b3b3b3b3b3b3b3</sha1>
</fingerprints>
</video>
<deletedVideo>
<member id="42">Example Member</member>
<id>video4</id>
<timestamp>2017-10-24T15:20:00Z</timestamp>
</deletedVideo>
</videos>
</queryResult>If no entries are found within the specified date range, the <queryResult> element will be empty and have a count of 0, as shown below.
<?xml version="1.0" encoding="UTF-8"?>
<queryResult xmlns="https://hashsharing.ncmec.org/hashsharing/v2">
<images count="0" />
<videos count="0" />
</queryResult>7.1.1. Querying Entries with Parameters
The following is a series of examples of query parameters. These can be combined with one another.
Querying by Submission Type
Entry queries can also be limited to a specific type.
The same requirements for parameters still apply when querying by type.
For example, to query for only videos submitted or last updated between 2017-10-20 00:00:00Z and 2017-10-30 00:00:00Z, the following curl call can be used.
curl --user usr123:pswd123 \
--data-urlencode "from=2017-10-201T00:00:00.000Z" \
--data-urlencode "to=2017-10-30T00:00:00.000Z" \
--get https://hashsharing-test.ncmec.org/npo/v2/entries/videosNote the addition of /videos to the endpoint URL.
Similarly, a query for only image entries can be made by using the endpoint /v2/entries/images.
The response format is the same as when querying for entries of all types, except that only entries of the requested type will be included.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<queryResult xmlns="https://hashsharing.ncmec.org/hashsharing/v2">
<images count="0"/>
<videos count="4" maxTimestamp="2017-10-24T15:20:00Z">
<video>
<member id="42">Example Member</member>
<timestamp>2017-10-24T15:00:00Z</timestamp>
<id>video1</id>
<fingerprints>
<md5>b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1</md5>
<sha1>b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1</sha1>
</fingerprints>
<categorizations>
<categorization guid="01234567-abcd-0123-4567-0123456789ab" name="Example Categorization 1"/>
<categorization guid="01255555-abcd-0123-4567-0123456789cd" name="Example Categorization 2"/>
</categorizations>
<feedback lastUpdateTimestamp="2023-06-01T12:48:14Z">
<affirmativeFeedback type="Sha1" lastUpdateTimestamp="2023-06-01T12:48:14Z">
<members timestamp="2023-06-01T12:48:14Z">
<member id="42">Example Member</member>
</members>
</affirmativeFeedback>
</feedback>
</video>
<video>
<member id="42">Example Member</member>
<timestamp>2017-10-24T15:00:00Z</timestamp>
<id>video2</id>
<fingerprints>
<md5>b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2</md5>
<sha1>b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2</sha1>
</fingerprints>
<categorizations />
<feedback />
</video>
<video>
<member id="42">Example Member</member>
<timestamp>2017-10-24T15:00:00Z</timestamp>
<id>video3</id>
<fingerprints>
<md5>b3b3b3b3b3b3b3b3b3b3b3b3b3b3b3b3</md5>
<sha1>b3b3b3b3b3b3b3b3b3b3b3b3b3b3b3b3b3b3b3b3</sha1>
</fingerprints>
<categorizations />
<feedback />
</video>
<deletedVideo>
<member id="42">Example Member</member>
<id>video4</id>
<timestamp>2017-10-24T15:20:00Z</timestamp>
</deletedVideo>
</videos>
</queryResult>Querying By Submitting Member
Entry query results can be limited to a specific submitting member.
To do this, specify the member ID of the submitting member using the query parameter memberId.
In this example, we will query for all entries by the member "Example Member".
To get a list of members and their member IDs, make a GET request to https://hashsharing-test.ncmec.org/npo/v2/members.
The following is the result:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<members xmlns="https://hashsharing.ncmec.org/hashsharing/v2" count="3">
<members id="1">Member 1</members>
<members id="2">Member 2</members>
<members id="42">Example Member</members>
</members>To query for only entries submitted by "Example Member" with ID 42, add memberId=42 to the query in addition to the required HTTP parameters.
curl --user usr123:pswd123 \
--data-urlencode "from=2017-10-201T00:00:00.000Z" \
--data-urlencode "to=2017-10-30T00:00:00.000Z" \
--data-urlencode "memberId=42" \
--get https://hashsharing-test.ncmec.org/npo/v2/entriesThe response format is the same as when querying for all entries except that only entries submitted by the specified member will be included.
7.1.2. Pagination
For large result sets, query responses may be paginated.
Results are paged when the <paging> element appears in the query result, and the next page of results can be retrieved by using the provided URL.
Paging URLs are relative to the appropriate application root from the beginning of this document.
| Only forward paging is supported and that the format of paging URLs provided by the API are subject to change without notice. Your implementation should not store paging URLs, should not assume that they will be valid in the future, and should not derive meaning from them. |
For example, assume a query is made that returns a large result set and that the user has already requested the second page of results. The response might look like the following:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<queryResult xmlns="https://hashsharing.ncmec.org/hashsharing/v2">
<images count="500" maxTimestamp="2017-10-24T16:00:00Z">
<image>
<member id="42">Example Member</member>
<timestamp>2017-10-24T16:00:00Z</timestamp>
<id>image1</id>
<classification>A1</classification>
<fingerprints>
<md5>a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1</md5>
<sha1>a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1</sha1>
<pdna>a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1...</pdna>
</fingerprints>
<categorizations>
<categorization guid="01234567-abcd-0123-4567-0123456789ab" name="Example Categorization 1"/>
<categorization guid="01255555-abcd-0123-4567-0123456789cd" name="Example Categorization 2"/>
</categorizations>
</image>
...
</images>
<videos count="500" maxTimestamp="2017-10-24T16:00:00Z">
<video>
<member id="42">Example Member</member>
<timestamp>2017-10-24T16:00:00Z</timestamp>
<id>video1</id>
<fingerprints>
<md5>b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1</md5>
<sha1>b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1</sha1>
</fingerprints>
<categorizations>
<categorization guid="01234567-abcd-0123-4567-0123456789ab" name="Example Categorization 1"/>
<categorization guid="01255555-abcd-0123-4567-0123456789cd" name="Example Categorization 2"/>
</categorizations>
</video>
...
</videos>
<paging>
<next>/v2/entries?from=2017-10-20T00%3A00%3A00.000Z&to=2017-10-30T00%3A00%3A00.000Z&start=2001&size=1000&max=3000</next>
</paging>
</queryResult>Note the <paging> element and its sub-element <next>.
To request the third page of results, the user would use the following curl command.
curl --user usr123:pswd123 \
https://hashsharing-test.ncmec.org/npo/v2/entries?from=2017-10-20T00%3A00%3A00.000Z&to=2017-10-30T00%3A00%3A00.000Zstart=2001&size=1000&max=30007.2. Submitting Entries
Entries are submitted as an XML document with <submission> as the root element.
The server responds with a summary of results.
| Large fingerprints are not submitted in this manner and must be done as part of a separate request. See the section on working with large fingerprints for more details. |
Adding categorizations to entries on submission is not supported. These will need to be done after in a separate HTTP request, see categorizations.
In this example, assume the following file is located on the file system at submission.xml.
<?xml version="1.0" encoding="UTF-8"?>
<submission xmlns="https://hashsharing.ncmec.org/hashsharing/v2">
<images>
<image>
<id>aaa</id>
<classification>A1</classification>
<fingerprints>
<md5>aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa</md5>
<sha1>aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa</sha1>
<pdna>aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...</pdna>
</fingerprints>
</image>
<image>
<id>bbb</id>
<classification>A2</classification>
<fingerprints>
<md5>bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb</md5>
<sha1>bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb</sha1>
<pdna>bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb...</pdna>
</fingerprints>
</image>
<image>
<id>ccc</id>
<classification>B1</classification>
<fingerprints>
<md5>cccccccccccccccccccccccccccccccc</md5>
<sha1>cccccccccccccccccccccccccccccccccccccccc</sha1>
<pdna>cccccccccccccccccccccccccccccccccccccccccccccccccccccc...</pdna>
</fingerprints>
</image>
<image>
<id>ddd</id>
<classification>B2</classification>
<fingerprints>
<md5>dddddddddddddddddddddddddddddddd</md5>
<sha1>dddddddddddddddddddddddddddddddddddddddd</sha1>
<pdna>dddddddddddddddddddddddddddddddddddddddddddddddddddddd...</pdna>
</fingerprints>
</image>
</images>
<videos>
<video>
<id>zzz</id>
<fingerprints>
<md5>ffffffffffffffffffffffffffffffff</md5>
<sha1>ffffffffffffffffffffffffffffffffffffffff</sha1>
</fingerprints>
</video>
<video>
<id>yyy</id>
<fingerprints>
<md5>eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee</md5>
<sha1>eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee</sha1>
</fingerprints>
</video>
</videos>
</submission>The curl call to perform the submission is shown below.
curl --user usr123:pswd123 \
--header 'Content-Type: application/xml; charset=utf-8' \
--data @submission.xml \
https://hashsharing-test.ncmec.org/npo/v2/entriesThe server will respond with an XML message indicating the number of entries that were received, accepted, new, and updated, grouped by entry type.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<submissionResult xmlns="https://hashsharing.ncmec.org/hashsharing/v2">
<images>
<received>4</received>
<accepted>4</accepted>
<new>4</new>
<updated>0</updated>
</images>
<videos>
<received>2</received>
<accepted>2</accepted>
<new>2</new>
<updated>0</updated>
</videos>
</submissionResult>7.3. Updating Entries
The same process for submitting entries is used to update entries. An entry can be updated by resubmitting it with new data but reusing an existing member-specified entry ID. As such, entries may only be updated by the member that originally submitted them.
| While some fingerprints can be removed in this manner by omitting them from the update payload, large fingerprints must be removed individually using the endpoints for each fingerprint type. |
Assume the file submission.xml below is submitted to Hash Sharing.
<?xml version="1.0" encoding="UTF-8"?>
<submission xmlns="https://hashsharing.ncmec.org/hashsharing/v2">
<images>
<image>
<id>a</id>
<classification>A1</classification>
<fingerprints>
<pdna>aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...</pdna>
</fingerprints>
</image>
<image>
<id>b</id>
<classification>B1</classification>
<fingerprints>
<pdna>b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1...</pdna>
</fingerprints>
</image>
</images>
</submission>The server will respond with the following result.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<submissionResult xmlns="https://hashsharing.ncmec.org/hashsharing/v2">
<images>
<received>2</received>
<accepted>2</accepted>
<new>2</new>
<updated>0</updated>
</images>
<videos>
<received>0</received>
<accepted>0</accepted>
<new>0</new>
<updated>0</updated>
</videos>
</submissionResult>Querying for these entries returns the following result.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<queryResult xmlns="https://hashsharing.ncmec.org/hashsharing/v2">
<images count="2" maxTimestamp="2017-10-24T14:00:00Z">
<image>
<member id="42">Example Member</member>
<timestamp>2017-10-24T14:00:00Z</timestamp>
<id>a</id>
<classification>A1</classification>
<fingerprints>
<pdna>aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...</pdna>
</fingerprints>
</image>
<image>
<member id="42">Example Member</member>
<timestamp>2017-10-24T14:00:00Z</timestamp>
<id>b</id>
<classification>B1</classification>
<fingerprints>
<pdna>b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1...</pdna>
</fingerprints>
</image>
</images>
<videos count="0" />
</queryResult>Now assume another file update.xml exists, below.
Notice that the entry ID b appears in both submission.xml and update.xml, but that the data for the entry identified by ID b is different in update.xml: the PhotoDNA fingerprint has changed and the entry now includes an MD5 fingerprint.
Also included in update.xml is a new entry with ID c.
<?xml version="1.0" encoding="UTF-8"?>
<submission xmlns="https://hashsharing.ncmec.org/hashsharing/v2">
<images>
<image>
<id>b</id>
<classification>B2</classification>
<fingerprints>
<pdna>b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2...</pdna>
<md5>b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2</md5>
</fingerprints>
</image>
<image>
<id>c</id>
<classification>A2</classification>
<fingerprints>
<pdna>cccccccccccccccccccccccccccccccccccccccccccccccccccccc...</pdna>
</fingerprints>
</image>
</images>
</submission>This file is submitted by the same user that submitted submission.xml, and the server responds with the following result.
Note that one image entry is new, and one image entry is updated.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<submissionResult xmlns="https://hashsharing.ncmec.org/hashsharing/v2">
<images>
<received>2</received>
<accepted>2</accepted>
<new>1</new>
<updated>1</updated>
</images>
<videos>
<received>0</received>
<accepted>0</accepted>
<new>0</new>
<updated>0</updated>
</videos>
</submissionResult>Querying now produces the following result.
Notice that there are now three entries (with IDs a, b, and c), and that the data for the entry identified by member ID 42 and entry ID b is that from update.xml.
Note also that timestamp for the entry identified by member ID 42 and entry ID b is later than it was in the first query result.
<?xml version="1.0" encoding="UTF-8"?>
<queryResult xmlns="https://hashsharing.ncmec.org/hashsharing/v2">
<images count="3" maxTimestamp="2017-10-24T14:30:00Z">
<image>
<member id="42">Example Member</member>
<timestamp>2017-10-24T14:00:00</timestamp>
<id>a</id>
<classification>A1</classification>
<fingerprints>
<pdna>aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...</pdna>
</fingerprints>
</image>
<image>
<member id="42">Example Member</member>
<timestamp>2017-10-24T14:30:00</timestamp>
<id>b</id>
<classification>B2</classification>
<fingerprints>
<md5>b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2</md5>
<pdna>b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2...</pdna>
</fingerprints>
</image>
<image>
<member id="42">Example Member</member>
<timestamp>2017-10-24T14:30:00Z</timestamp>
<id>c</id>
<classification>A2</classification>
<fingerprints>
<pdna>cccccccccccccccccccccccccccccccccccccccccccccccccccccc...</pdna>
</fingerprints>
</image>
</images>
<videos count="0" />
</queryResult>7.4. Entry Validation Errors
There are two types of error conditions that may occur when submitting and updating entries: structural errors and data errors.
7.4.1. Structural Errors
Structural errors occur when the structure of the XML document does not conform to the schema. When the server detects a structural error, the entire submission is rejected.
Shown below is an example of a file with a structural error.
Note that the root element is <causesError> instead of <submission>, an error with the structure of the XML document.
<?xml version="1.0" encoding="UTF-8"?>
<causesError xmlns="https://hashsharing.ncmec.org/hashsharing/v2">
<images>
<image>
<id>image1</id>
<classification>A1</classification>
<fingerprints>
<md5>a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1</md5>
<sha1>a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1</sha1>
<pdna>a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1...</pdna>
</fingerprints>
</image>
</images>
<videos>
<video>
<id>video1</id>
<fingerprints>
<md5>b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1</md5>
<sha1>b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1b1</sha1>
</fingerprints>
</video>
</videos>
</causesError>If this file is submitted, the server will respond with an error message, as shown below, and all entries will be rejected by the system.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<error xmlns="https://hashsharing.ncmec.org/hashsharing/v2">
<requestId>f2a98634-1aea-4478-b990-ea03b19d8147</requestId>
<code>4110</code>
<status>Malformed XML submittal</status>
<message>unexpected element (uri:"https://hashsharing.ncmec.org/hashsharing/v2", local:"causesError").
Expected elements are <{https://hashsharing.ncmec.org/hashsharing/v2}submission></message>
</error>A similar message will be returned for any submitted XML that is not well-formed or cannot properly be mapped to the specification provided in schema.xsd.
7.4.2. Data Errors
Data errors occur when the XML document is properly structured but the format of the data itself is invalid. When a data error is detected, only those entries that contain invalid data are rejected. Entries that pass validation are still accepted. Each offending entry is rejected in its entirety, no data that is rejected is saved into the Hash Sharing system, and an error message is provided to the user to indicate the cause of the error.
Shown below is an example of a submission with a validation error.
Note that for the first image entry, the classification is ERROR1 and the PhotoDNA fingerprint is ERROR2, neither of which conforms to the data specification.
Likewise, for the first video entry, the MD5 fingerprint is ERROR3 and the SHA1 fingerprint is ERROR4.
<?xml version="1.0" encoding="UTF-8"?>
<submission xmlns="https://hashsharing.ncmec.org/hashsharing/v2">
<images>
<image>
<id>image1</id>
<classification>ERROR1</classification>
<fingerprints>
<md5>a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1</md5>
<sha1>a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1</sha1>
<pdna>ERROR2</pdna>
</fingerprints>
</image>
<image>
<id>image2</id>
<classification>A2</classification>
<fingerprints>
<md5>a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2</md5>
<sha1>a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2</sha1>
<pdna>a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2a2...</pdna>
</fingerprints>
</image>
</images>
<videos>
<video>
<id>video1</id>
<fingerprints>
<md5>ERROR3</md5>
<sha1>ERROR4</sha1>
</fingerprints>
</video>
<video>
<id>video2</id>
<fingerprints>
<md5>b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2</md5>
<sha1>b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2b2</sha1>
</fingerprints>
</video>
</videos>
</submission>If this file is submitted, the system will reject the first image entry and first video entry but accept the second image and video entries. The server will respond with a message indicating that two image entries and two video entries were received, and one new entry was accepted for each. The response will also include the original data of the rejected entries alongside validation error messages.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<submissionResult xmlns="https://hashsharing.ncmec.org/hashsharing/v2">
<images>
<received>2</received>
<accepted>1</accepted>
<new>1</new>
<updated>0</updated>
<rejected count="1">
<image>
<id>image1</id>
<classification>ERROR1</classification>
<fingerprints>
<md5>a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1</md5>
<sha1>a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1</sha1>
<pdna>ERROR2</pdna>
</fingerprints>
<error element="classification">invalid classification</error>
<error element="fingerprints.pdna">must match "[a-fA-F0-9]{288}"</error>
</image>
</rejected>
</images>
<videos>
<received>2</received>
<accepted>1</accepted>
<new>1</new>
<updated>0</updated>
<rejected count="1">
<video>
<id>video1</id>
<fingerprints>
<md5>ERROR3</md5>
<sha1>ERROR4</sha1>
</fingerprints>
<error element="fingerprints.sha1">must match "[a-fA-F0-9]{40}"</error>
<error element="fingerprints.md5">must match "[a-fA-F0-9]{32}"</error>
</video>
</rejected>
</videos>
</submissionResult>7.5. Retracting Entries
There are two ways to retract entries: individually per request or multiple in the same request.
7.5.1. Individual Entry
To retract an individual entry, make a DELETE request to the endpoint https://{applicationRoot}/v2/entries/{entryId}, where {entryId} is the entry ID of the entry to retract.
Entries may only be retracted by the member that submitted them.
For example, to retract entry ID bbb, use the following curl call.
curl --user usr123:pswd123 --request DELETE https://hashsharing-test.ncmec.org/npo/v2/entries/bbbIf the entry was successfully retracted, the server responds with the following:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<deletionResult xmlns="https://hashsharing.ncmec.org/hashsharing/v2">
<deleted>1</deleted>
</deletionResult>If the entry did not exist or was not retracted for any other reason, the <deleted> element will have a value of 0.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<deletionResult xmlns="https://hashsharing.ncmec.org/hashsharing/v2">
<deleted>0</deleted>
</deletionResult>To make other users aware that an entry has been retracted, query results include a record of the retraction with the unique key of the entry (the submitting member and the member-specified entry ID) and the timestamp of when the entry was retracted.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<queryResult xmlns="https://hashsharing.ncmec.org/hashsharing/v2">
<images count="1" maxTimestamp="2017-10-24T15:00:00Z">
<deletedImage>
<member id="42">Example Member</member>
<id>bbb</id>
<timestamp>2017-10-24T15:00:00Z</timestamp>
</deletedImage>
</images>
<videos count="1" maxTimestamp="2017-10-24T15:30:00Z">
<deletedVideo>
<member id="42">Example Member</member>
<id>ccc</id>
<timestamp>2017-10-24T15:00:00Z</timestamp>
</deletedVideo>
</videos>
</queryResult>7.5.2. Multiple Entries
To retract multiple entries in a single request, make a DELETE request to the endpoint https://{applicationRoot}/v2/entries/ with a body containing a <deletionRequest> XML document.
Assume the following file is located on the file system at retraction.xml.
<?xml version="1.0" encoding="UTF-8"?>
<deletionRequest xmlns="https://hashsharing.ncmec.org/hashsharing/v2">
<ids>
<id>aaa</id>
<id>bbb</id>
<id>ccc</id>
</ids>
</deletionRequest>The curl call to retract the above entries is show below.
curl --user usr123:pswd123 \
--header 'Content-Type: application/xml; charset=utf-8' \
--data @retraction.xml \
--request DELETE https://hashsharing-test.ncmec.org/npo/v2/entriesThe server responds with the following, which includes a count of the number of actually retracted entries.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<deletionResult xmlns="https://hashsharing.ncmec.org/hashsharing/v2">
<deleted>3</deleted>
</deletionResult>Entries meeting the following criteria will be skipped and will not be added to this total:
1. Cannot be found for the current user.
2. Of the wrong type, i.e. a video when the user used the images endpoint https://{applicationRoot}/v2/entries/images for retraction.
3. Have already been retracted.
Below is a result where one of the elements match the above criteria, thus only deleting 2 elements.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<deletionResult xmlns="https://hashsharing.ncmec.org/hashsharing/v2">
<deleted>2</deleted>
</deletionResult>As with individual retraction, to make other users aware that an entry has been retracted, query results include a record of the retraction with the unique key of the entry (the submitting member and the member-specified entry ID) and the timestamp of when the entry was retracted.
7.6. Uploading Fingerprint Directly
In addition to submitting all small fingerprints directly via the submission api, it is also possible to update them individually via uploading files. For large fingerprints this is the ONLY way to upload them.
7.6.1. Fingerprint Types
Below is a list of fingerprints supported via this endpoint.
7.6.2. Submitting a Fingerprint via File
A fingerprint can be added to an existing entry using a PATCH request to the endpoint https://{applicationRoot}/v2/entries/{entryId}/fingerprints/{fingerprintType}, where {fingerprintType} is the key for a fingerprint types.
The server will respond with a status of 201 Created if no existing fingerprint of this type had been submitted for the specified Entry ID.
If a fingerprint already exists for the specified type, the response status will be 204 No Content.
The existing fingerprint will be replaced with the one supplied in the request.
For example, to add or replace a Videntifier fingerprint for an existing entry with Entry ID video1, make the following curl call.
In this example, video1.desc72 is a file on disk containing the raw bytes for the Videntifier fingerprint.
curl --user usr123:pswd123 \
--header 'Content-Type: application/octet-stream' \
--data-binary @file.desc72 \
--request PATCH \
https://hashsharing-test.ncmec.org/npo/v2/entries/video1/fingerprints/VIDENTIFIER
Hash Sharing does not support multipart/form-data file uploads.
Content-Type and Content-Length headers must be included for file uploads.
|
7.7. Working with Large Fingerprints
Fingerprints that are too large to be serialized as strings in request and response payloads are instead handled as files with dedicated endpoints for upload, removal, and retrieval, described below.
| Your implementation should not store fingerprint retrieval URLs, should not assume that they will be valid in the future, and should not derive meaning from them. |
7.7.1. Format
Fingerprints submitted using the large fingerprint endpoint MUST be submitted as raw binary data. These fingerprints should not be transmitted in any format that does not meet the conventions for the fingerprint type.
7.7.2. Supported Large Fingerprint Types
Below is a list of fingerprints only supported via these endpoints. Fingerprint values for these types cannot be included in XML submission documents and are not included in query responses.
| Name | Key |
|---|---|
Videntifier |
|
TMK+PDQF |
|
SSVH+PDNA |
|
SSVH+SaferHash |
|
7.7.3. Discovering Large Fingerprints
Due to their size, file-based fingerprints are not serialized in the response payload when querying for entries. Instead, the response includes a relative URL that can be appended to the application’s root URL to retrieve the fingerprint.
The response below is an example of how file-based fingerprints are presented when querying for entries.
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<queryResult xmlns="https://hashsharing.ncmec.org/hashsharing/v2">
<images count="2" maxTimestamp="2017-10-24T14:00:00Z">
<image>
<member id="42">Example Member</member>
<timestamp>2017-10-24T14:00:00Z</timestamp>
<id>image1</id>
<classification>A1</classification>
<fingerprints>
<pdna>aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...</pdna>
<videntifier rel="self" href="/v2/image1VidentifierUrl" />
</fingerprints>
</image>
</images>
<videos count="1" maxTimestamp="2017-10-24T14:00:00Z">
<video>
<member id="42">Example Member</member>
<timestamp>2017-10-24T14:00:00Z</timestamp>
<id>video1</id>
<classification>B1</classification>
<fingerprints>
<md5>0123456789abcdef0123456789abcdef</pdna>
<tmk-pdqf rel="self" href="/v2/video1TmkPdqfUrl" />
<videntifier rel="self" href="/v2/video1VidentifierUrl" />
<ssvh-pdna rel="self" href="/v2/video1SsvhPdnaUrl" />
<ssvh-safer-hash rel="self" href="/v2/video1SsvhSaferHashUrl" />
</fingerprints>
</video>
</videos>
</queryResult>| That the relative fingerprint URLs in this response are illustrative and are not representative of real data. When interacting with the API, these URLs should be treated as opaque and are subject to change without notice. |
7.7.4. Retrieving a Large Fingerprint
A large fingerprint can be retrieved with a GET request to the URL supplied in the query result resolved against the application root URL, i.e. https://{applicationRoot}/{fingerprintUrl}.
The response is the raw binary content of the fingerprint.
For example, to retrieve the TMK+PDQF fingerprint for the video in the query result from Discovering Large Fingerprints, make the following curl call.
curl --user usr123:pswd123 https://hashsharing-test.ncmec.org/npo/v2/video1TmkPdqfUrl7.7.5. Submitting and Updating a Large Fingerprint
In addition to being able to upload small fingerprints as binaries, one can also upload larger fingerprints. See Submitting a Fingerprint via File.
7.7.6. Deleting a Large Fingerprint
A large fingerprint can be removed using a DELETE request to the endpoint https://{applicationRoot}/v2/entries/{entryId}/fingerprints/{fingerprintType}, where {fingerprintType} is the key for a supported large fingerprint type.
The server will respond with a status of 204 No Content if a fingerprint existed and was deleted.
For example, to remove a TMK+PDQF fingerprint for an existing entry with Entry ID video1, make the following curl call.
curl --user usr123:pswd123 \
--request DELETE \
https://hashsharing-test.ncmec.org/npo/v2/entries/video1/fingerprints/TMK_PDQF7.8. Adding Categorizations to Entries
It is possible to annotate your entries with categorizations to better contextualize their content, context, etc.
| The categorization IDs provided here are for example only. Different instances of hash sharing will have unique IDs i.e. CSAM versus Exploitative. Though for each instance, individual categorizations' IDs are static. |
7.8.1. Listing Available Categorizations.
To list all categorizations available to annotate entries with, make a GET request to https://hashsharing-test.ncmec.org/npo/v2/categorizations.
| The IDs for each categorization will not change, however; there may be more added in the future. For this reason, it is safe to cache these values, though keep in mind different environments will contain unique IDs. i.e. Known Minor on CSAM will have a different ID than Exploitative. |
The curl call to execute this is shown below.
curl --user usr123:pswd123 \
--get https://hashsharing-test.ncmec.org/npo/v2/categorizationsThe response is a list of active categorizations.
| The IDs below are not reflective of actual data, they are provided as example only. |
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<availableCategorizations xmlns="https://hashsharing.ncmec.org/hashsharing/v2">
<group name="Content" allowMultiple="false">
<categorization guid="01234567-abcd-0123-4567-012345678900" name="Clothed Child"/>
<categorization guid="01234567-abcd-0123-4567-012345678901" name="Unclothed Child"/>
<categorization guid="01234567-abcd-0123-4567-012345678902" name="Unconfirmed Child Sexual Abuse Material"/>
</group>
<group name="Context" allowMultiple="true">
<categorization guid="01234589-abcd-0123-4567-012345678910" name="Known Minor"/>
<categorization guid="01234577-abcd-0123-4567-012345678911" name="Potential Meme"/>
<categorization guid="01234566-abcd-0123-4567-012345678912" name="Viral"/>
</group>
<group name="Egregious" allowMultiple="true">
<categorization guid="01234444-abcd-0123-4567-012345678920" name="Bestiality"/>
</group>
</availableCategorizations>7.8.2. Current Available Categorizations
| Categorizations may be added at a later date. It is possible that the list below is not up-to-date with what is present. What is provided below is for your convenience: |
Group |
Name |
Content |
Varies for CSAM versus Exploitative |
Context |
Known Minor |
Potential Meme |
|
Viral |
|
Egregious |
Bestiality |
Bondage/Sadism |
|
Infant/Toddler |
7.8.3. Assigning Categorizations
With the categorizations defined above, you are able to assign them to entries you’ve already created.
| The following request will override any existing categorizations with the list you supply. If you’d like to keep existing categorizations, you must include them in the PUT request. |
Assume the file categorization-update.xml below is submitted to Hash Sharing.
<?xml version="1.0" encoding="UTF-8"?>
<categorizationAssignmentUpdateRequest xmlns="https://hashsharing.ncmec.org/hashsharing/v2">
<ids>
<guid>836e9e04-3443-11ed-a261-0242ac120002</guid>
</ids>
</categorizationAssignmentUpdateRequest>The curl call to perform update categorizations for an entry is shown below.
curl --user usr123:pswd123 \
--header 'Content-Type: application/xml; charset=utf-8' \
--data @categorization-update.xml \
--request PUT https://hashsharing-test.ncmec.org/npo/v2/entries/{memberEntryId}/categorizations7.9. Adding Feedback to Entries
It is possible to add feedback to an entry based on its fingerprint(s), to provide better context for a fingerprint.
| The feedback reason IDs provided here are for example only. |
7.9.1. Identifying the FeedbackType
When working with feedback, the fingerprint is now categorized as a feedbackType, the available fingerprints can be seen here.
7.9.2. Listing Available Downvote Reasons.
Feedback is split between affirmative (upvote) or negative (downvote) reasons, and only the downvote reasons are provided.
To list all available downvote reasons for a given feedbackType, make a GET request to https://hashsharing-test.ncmec.org/npo/v2/feedback/PDNA/reasons.
| The IDs for each feedback reasons will not change, however; there may be more added in the future. For this reason, it is safe to cache these values. |
The curl command to execute this request is shown below:
curl --user usr123:pswd123 \
--get https://hashsharing-test.ncmec.org/npo/v2/feedback/{feedbackType}/reasonsThe response is a list of available downvote reasons.
| The IDs below are not reflective of actual data, they are provided as example only. |
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<availableFeedbackReasons xmlns="https://hashsharing.ncmec.org/hashsharing/v2">
<reason guid="01234567-abcd-0123-4567-012345678900" name="Example Reason 1" type="Sha1"/>
</availableFeedbackReasons>7.9.3. Providing Feedback for an Entry
Providing feedback is split between Upvote or Downvote schemas, and only 1 can be provided at a time.
| A member can only submit feedback for a single schema and feedbackType, at a time. |
| Providing the same feedback as before will not update the record, or the timestamp. |
Providing Affirmative Feedback (Upvotes) for an Entry
Providing upvotes is simple because it doesn’t require any feedback reason IDs in the submission, as they are automatically assigned later.
Assume the file affirmative-submission.xml below is submitted to Hash Sharing.
<?xml version="1.0" encoding="UTF-8"?>
<feedbackSubmission xmlns="https://hashsharing.ncmec.org/hashsharing/v2">
<affirmative>
<!-- Intentionally left blank -->
</affirmative>
</feedbackSubmission>| The current implementation allows the upvote submission to be empty, but that is subject to change. |
The curl command to execute this request is shown below:
curl --user usr123:pswd123 \
--header 'Content-Type: application/xml; charset=utf-8' \
--data @affirmative-submission.xml \
--request PUT https://hashsharing-test.ncmec.org/npo/v2/entries/{memberId}/{memberEntryId}/{feedbackType}/feedback| To update this feedback, submit a downvote request. |
Providing Negative Feedback (Downvotes) for an Entry
Before a member can provide a downvote, they need to query the available downvote reason shown here, so they can attach a feedback reason ID to the feedback submission.
Assume the file negative-submission.xml below is submitted to Hash Sharing.
<?xml version="1.0" encoding="UTF-8"?>
<feedbackSubmission xmlns="https://hashsharing.ncmec.org/hashsharing/v2">
<negative>
<reasonIds>
<guid>01234567-abcd-0123-4567-012345678900</guid>
</reasonIds>
</negative>
</feedbackSubmission>The curl command to execute this request is shown below:
curl --user usr123:pswd123 \
--header 'Content-Type: application/xml; charset=utf-8' \
--data @negative-submission.xml \
--request PUT https://hashsharing-test.ncmec.org/npo/v2/entries/{memberId}/{memberEntryId}/{feedbackType}/feedback| To update this feedback, submit an upvote request, or resubmit the downvote request with a different reason ID. |
7.10. Controlling Access to Entries
By default, any entry you add will be accessible to any other member. You may add restrictions by specifying which members should have access to your entries using the API calls shown in this section.
To get a list of members and their member IDs, make a GET request to https://{applicationRoot}/v2/members.
7.10.1. Listing Current Grants
To get a list of members to whom you have granted explicit access, make a GET request to https://{applicationRoot}/v2/permissions/access.
If no members are included in the response, then you have granted no explicit access and ALL members may see your entries.
7.10.2. Granting Access
To grant access to your entries to another member, make a POST request to https://{applicationRoot}/v2/permissions/access/{memberId}, replacing {memberId} with the ID of the member for whom you are granting access.
To remove access from all members except yourself, specify only your own member ID using this endpoint.
7.10.3. Revoking Access
To remove access to your entries from another member, make a DELETE request to https://{applicationRoot}/v2/permissions/access/{memberId}, replacing {memberId} with the ID of the member for whom you are revoking access.
If you remove the last permitted member, ALL members will be able to access your entries.
Revoking access from a member for whom you have not previously granted explicit access has no effect.
8. Troubleshooting
If you encounter problems using this web service, the following steps may help you with troubleshooting.
8.1. Authentication Errors
If the client does not specify authentication when making a request, the following error message will be returned, and the HTTP status code of the response will be 401 Unauthorized.
The request ID will be unique to your request.
<?xml version="1.0" encoding="UTF-8"?>
<error xmlns="https://hashsharing.ncmec.org/hashsharing/v2">
<requestId>7a477191-7e23-4bf9-883d-c0a5efcbefe1</requestId>
<code>2000</code>
<status>Authentication required</status>
<message>Full authentication is required to access this resource</message>
</error>8.2. Authorization Errors
If the client does not have permission to perform the requested action, the following error message will be returned, and the HTTP status code of the response will be 403 Forbidden.
The request ID will be unique to your request.
<?xml version="1.0" encoding="UTF-8"?>
<error xmlns="https://hashsharing.ncmec.org/hashsharing/v2">
<requestId>a7bb4497-c4d4-42a0-8c56-6d7f48a58793</requestId>
<code>3000</code>
<status>User is not authorized to perform this action</status>
<message>Access is denied</message>
</error>8.3. Status Endpoint
Verify that you can access the server and that your credentials are correct by accessing the status endpoint from a web browser.
For example, to do this in the NPO CSAM test environment, navigate to https://hashsharing-test.ncmec.org/npo/v2/status.
Your browser will prompt you to provide credentials.
Verify that you receive the response below, where {ipAddress} is your IP address, {username} is your username, and {member} is the name of your organization.
<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<status xmlns="https://hashsharing.ncmec.org/hashsharing/v2">
<ipAddress>{ipAddress}</ipAddress>
<username>{username}</username>
<member id="{memberId}">{member}</member>
</status>If you receive the above response, you know that you can access the server and authenticate successfully. If you cannot, contact NCMEC to verify your credentials. Verify that you do not have any firewall rules that prevent you from connecting to NCMEC’s servers.
8.4. Timestamp Format
Date/time values must be specified in the ISO 8601 format yyyy-MM-dd’T’HH:mm:ss.SSSZ and must be URL-encoded when used in query parameters.
ISO 8601 timestamps must have a time zone specified in the format +/-hh:mm or Z for UTC.
Appendix A: Schema
Below is a description of element types specified by the Hash Sharing schema.
Listed with each element or attribute is an indicator of the element’s or attribute’s cardinality.
| Cardinality | Description |
|---|---|
1 |
exactly one element is required |
1+ |
one or more elements is required |
0|1 |
element is optional but at most 1 element may be supplied |
0+ |
element is optional but there is no limit on the number that may be supplied |
In the event that a member chooses to voluntarily provide information for an element, certain parent, child, or sibling elements may also be required. Please refer to the XSD for the authoritative schema definition.
A.1. Common Types
<member>1-
The name of a Hash Sharing member.
type stringid1attribute-
The ID of the Hash Sharing member.
type long
<categorizationGroup>1-
A group of similar categorizations.
<categorization>1+-
Categorizations belonging to this group.
type<categorization> name1attribute-
The name of the group.
type string allowMultiple1attribute-
If the multiple categorizations are allowed from this group.
type boolean
<categorization>1-
An annotation detailing the content, context, etc. of an entry.
guid1attribute-
The GUID of the categorization.
type guid name1attribute-
The name of the categorization.
type string
<feedback>1-
The root element detailing the feedback of an entry’s fingerprint(s).
lastUpdateTimestamp1attribute-
The timestamp of when the feedback was last updated.
type ISO 8601 date and time
A.2. General Response Types
<status>-
The root element for details about the currently authenticated user.
<ipAddress>1-
The IP address of the currently authenticated user.
type string <username>1-
The username of the currently authenticated user.
type string <member>1-
The member details for the currently authenticated user.
type<member>
<error>-
The root element when an error occurs.
<requestId>-
The unique ID for the request that produced the error.
type string <code>-
An application error code.
type int <status>-
A description of the error.
type string <message>-
Additional details about the error.
type string
<members>-
Container element for a list of Hash Sharing members.
count1attribute-
The count of members in this list.
type int <member>-
A Hash Sharing member.
type<member>
A.3. Query Result Types
<queryResult>-
The root element of the response to a query for entries.
<images>0|1-
Container element for query result image entries.
count1attribute-
The count of image entries on the current page of results.
type int maxTimestamp0|1attribute-
The most recent update timestamp for image entries on the current page of results.
type ISO 8601 date and time <image>1+-
An image entry.
type<image> <deletedImage>1+-
An image entry that has been retracted.
type<deletedImage>
<videos>0|1-
Container element for query result video entries.
count1attribute-
The count of video entries on the current page of results.
type int maxTimestamp0|1attribute-
The most recent update timestamp for video entries on the current page of results.
type ISO 8601 date and time <video>1+-
A video entry.
type<video> <deletedVideo>1+-
A video entry that has been retracted.
type<deletedVideo>
<paging>0|1-
Container element for pagination URLs.
<previous>0|1-
The URL, relative to the application base URL, where the previous page of results can be retrieved.
Note that this functionality is not currently supported and this element is not used.type string <next>0|1-
The URL, relative to the application base URL, where the next page of results can be retrieved.
type string
A.3.1. Query Result Image
<image>-
A query result image.
<member>1-
The member that submitted this entry.
type<member> <timestamp>1-
The timestamp that this entry was submitted or last updated.
type ISO 8601 date and time <id>1-
The member-specified entry ID for this entry.
type string <classification>0|1-
The member-specified industry classification for this entry. One of:
-
A1
-
A2
-
B1
-
B2
type string -
<fingerprints>1-
Container elements for fingerprints for this entry.
<md5>0|1-
The member-specified MD5 fingerprint for this entry.
type string <sha1>0|1-
The member-specified SHA1 fingerprint for this entry.
type string <pdna>1-
The member-specified PDNA fingerprint for this entry.
type string <pdq>0|1-
The member-specified PDQ fingerprint for this entry.
type string <netClean>0|1-
The member-specified NetClean fingerprint for this entry.
type string <videntifier>0|1-
A hypermedia link to retrieve the member-specified Videntifier fingerprint for this entry.
rel1attribute-
The link relation to the fingerprint, always
self. href1attribute-
The URL, relative to the application base URL, where the fingerprint can be retrieved.
<categorizations>1-
Container elements for categorizations for this entry.
<categorization>0+-
A categorization.
type<categorization>
A.3.2. Retracted Image
<deletedImage>-
An image entry that has been retracted by the submitting member.
<member>1-
The member that submitted and retracted this entry.
type<member> <timestamp>1-
The timestamp that this entry was retracted.
type ISO 8601 date and time
A.3.3. Query Result Video
<video>-
A query result video.
<member>1-
The member that submitted this entry.
type<member> <timestamp>1-
The timestamp that this entry was submitted or last updated.
type ISO 8601 date and time <id>1-
The member-specified entry ID for this entry.
type string <classification>0|1-
The member-specified industry classification for this entry. One of:
-
A1
-
A2
-
B1
-
B2
type string -
<fingerprints>1-
Container elements for fingerprints for this entry.
<md5>1-
The member-specified MD5 fingerprint for this entry.
type string <sha1>0|1-
The member-specified SHA1 fingerprint for this entry.
type string <netClean>0|1-
The member-specified NetClean fingerprint for this entry.
type string <tmk-pdqf>0|1-
A hypermedia link to retrieve the member-specified TMK+PDQF fingerprint for this entry.
rel1attribute-
The link relation to the fingerprint, always
self. href1attribute-
The URL, relative to the application base URL, where the fingerprint can be retrieved.
<videntifier>0|1-
A hypermedia link to retrieve the member-specified Videntifier fingerprint for this entry.
rel1attribute-
The link relation to the fingerprint, always
self. href1attribute-
The URL, relative to the application base URL, where the fingerprint can be retrieved.
<ssvh-pdna>0|1-
A hypermedia link to retrieve the member-specified SSVH+PDNA fingerprint for this entry.
rel1attribute-
The link relation to the fingerprint, always
self. href1attribute-
The URL, relative to the application base URL, where the fingerprint can be retrieved.
<ssvh-safer-hash>0|1-
A hypermedia link to retrieve the member-specified SSVH+SaferHash fingerprint for this entry.
rel1attribute-
The link relation to the fingerprint, always
self. href1attribute-
The URL, relative to the application base URL, where the fingerprint can be retrieved.
<categorizations>1-
Container elements for categorizations for this entry.
<categorization>0+-
A categorization.
type<categorization>
A.3.4. Retracted Video
<deletedVideo>-
An video entry that has been retracted by the submitting member.
<member>1-
The member that submitted and retracted this entry.
type<member> <timestamp>1-
The timestamp that this entry was retracted.
type ISO 8601 date and time
A.4. Submission Request Types
<submission>-
The root element used to submit and update entries.
A.4.1. Image Submission
<image>-
An entry representing an image file.
<id>1-
The member-specified ID to uniquely identify this entry among the submitting member’s entries.
type stringvalidation must not be blankvalidation must be between 1 and 100 characters <classification>0|1-
Member-specified industry classification for this entry. One of:
-
A1
-
A2
-
B1
-
B2
type string -
<fingerprints>1-
Container element for one or more fingerprints for this image entry.
validation must contain at least one fingerprint<md5>0|1-
The MD5 fingerprint for this image entry.
type stringvalidation must conform to the regular expression[a-fA-F0-9]{32} <sha1>0|1-
The SHA1 fingerprint for this image entry.
type stringvalidation must conform to the regular expression[a-fA-F0-9]{40} <pdna>0|1-
The PhotoDNA fingerprint for this image entry.
type stringvalidation must conform to the regular expression[a-fA-F0-9]{288} <pdq>0|1-
The PDQ fingerprint for this image entry.
type stringvalidation must conform to the regular expression[a-fA-F0-9]{64} <netClean>0|1-
The NetClean fingerprint for this image entry.
type stringvalidation must conform to the regular expression[a-fA-F0-9]{40}
A.4.2. Video Submission
<video>-
An entry representing a video file.
<id>1-
The member-specified ID to uniquely identify this entry among the submitting member’s entries.
type stringvalidation must not be blankvalidation must be in between 1 and 100 characters <classification>0|1-
Member-specified industry classification for this entry. One of:
-
A1
-
A2
-
B1
-
B2
type string -
<fingerprints>1-
Container element for one or more fingerprints for this video entry.
validation must contain at least one fingerprint<md5>0|1-
The MD5 fingerprint for this video entry.
type stringvalidation must conform to the regular expression[a-fA-F0-9]{32} <sha1>0|1-
The SHA1 fingerprint for this video entry.
type stringvalidation must conform to the regular expression[a-fA-F0-9]{40} <netClean>0|1-
The NetClean fingerprint for this video entry.
type stringvalidation must conform to the regular expression[a-fA-F0-9]{40}
A.5. Submission Result Types
<submissionResult>-
The root element of the response to the submission of entries.
<images>0|1-
Container element details about submitted image entries.
<received>1-
The count of image entries included in the submission request.
<accepted>1-
The count of image entries that were valid and accepted.
<new>1-
The count of valid image entries that were new.
<updated>1-
The count of valid image entries that were updated.
<rejected>0|1-
Container element for rejected image entries.
count1attribute-
The count of image entries that were rejected.
<image>1+-
A rejected image entry.
type<image>
<videos>0|1-
Container element for details about submitted video entries.
<received>1-
The count of video entries included in the submission request.
<accepted>1-
The count of video entries that were valid and accepted.
<new>1-
The count of valid video entries that were new.
<updated>1-
The count of valid video entries that were updated.
<rejected>0|1-
Container element for rejected video entries.
count1attribute-
The count of video entries that were rejected.
<video>1+-
A rejected video entry.
type<video>
A.5.1. Rejected Image
<image>-
A rejected image entry.
<id>1-
The provided ID for the rejected image entry.
type string <classification>0|1-
The provided industry classification for the rejected image entry.
type string <fingerprints>1-
Container element for the provided fingerprints for the rejected image entry.
<md5>0|1-
The provided MD5 fingerprint for the rejected image entry.
type string <sha1>0|1-
The provided SHA1 fingerprint for the rejected image entry.
type string <pdna>1-
The provided PhotoDNA fingerprint for the rejected image entry.
type string <pdq>0|1-
The provided PDQ fingerprint for the rejected image entry.
type string <netClean>0|1-
The provided NetClean fingerprint for the rejected image entry.
type string
<error>1+-
The description of a validation error for the rejected image entry.
type stringelement1attribute-
The path to the element that did not pass validation checks.
type string
A.5.2. Rejected Video
<video>-
A rejected video entry.
<id>1-
The provided ID for the rejected video entry.
type string <classification>0|1-
The provided industry classification for the rejected video entry.
type string <fingerprints>1-
Container element for the provided fingerprints for the rejected video entry.
<md5>1-
The provided MD5 fingerprint for the rejected video entry.
type string <sha1>0|1-
The provided SHA1 fingerprint for the rejected video entry.
type string <netClean>0|1-
The provided NetClean fingerprint for the rejected video entry.
type string
<error>1+-
The description of a validation error for the rejected video entry.
type stringelement1attribute-
The path to the element that did not pass validation checks.
type string
A.6. Categorization Request Types
<updateCategorizationRequest>-
The root element of the request for a replacing all categorization for an entry.
<ids>1-
The container element for which categorizations to add.
<guid>0+-
A GUID corresponding to an available categorization.
type guid
A.7. Categorization Result Types
<availableCategorizations>-
The root element of the response to list available categorizations.
<groups>1-
Container element for groups which categorizations belong to.
<group>1+-
A group which a set categorizations belongs to.
A.8. Retraction Request Types
<deletionRequest>-
The root element of the request for a bulk retraction of previously submitted entries.
<ids>1-
Container element for a list of entry IDs.
<id>1+-
The member-specified entry ID of the entry to retract.
type string
A.9. Retraction Result Types
<deletionResult>-
The root element of the response to a retraction of previously submitted entries.
<deleted>1-
The count of actually deleted entries.
type int
A.10. Feedback Request Types
<feedbackSubmission>-
The root element of the request for a providing feedback for an entry.
<affirmative>0|1-
The container element for providing affirmative feedback (upvotes).
<negative>0|1-
The container element for providing negative feedback (downvotes).
<reasonIds>1-
The container element for available downvote reason GUIDs.
<guid>1+-
A GUID corresponding to an available downvote reason.
type guid
A.11. Feedback Result Types
A.11.1. Available Negative Feedback Reasons
<availableFeedbackReasons>-
The root element of the result for querying downvote reasons.
<reason>1-
The element for a downvote reason.
guid1attribute-
The GUID of the downvote reason.
type guid name1attribute-
The name of the downvote reason.
type string type1attribute-
The fingerprint that the downvote reason is for.
type string
A.11.2. Entry Feedback Result
<feedback>-
The root element that is the response of querying feedback for an entry.
type<feedback>lastUpdateTimestamp1attribute-
The timestamp of when this feedback was last updated.
type ISO 8601 date and time <affirmativeFeedback>0+-
The container element for upvotes of an entry’s fingerprint(s).
type1attribute-
The fingerprint that this upvote was provided for.
type string lastUpdateTimestamp1attribute-
The timestamp of when this upvote was last updated.
type ISO 8601 date and time<members>1-
A list of members who’ve provided this upvote.
timestamp1attribute-
The timestamp of when the most recent member provided the upvote.
type ISO 8601 date and time<member>1+:-
The member that provided this upvote.
type<member>
<negativeFeedback>0+-
The container element for downvotes of an entry’s fingerprint(s).
type1attribute-
The fingerprint that this downvote was provided for.
type string lastUpdateTimestamp1attribute-
The timestamp of when this downvote was last updated.
type ISO 8601 date and time <reasons>1-
Container element for a list of reasons for why this downvote was provided.
<reason>1+:-
The element for a downvote reason.
guid1attribute-
A GUID corresponding to this downvote reason.
type guid name1attribute-
The name of this downvote reason.
type string type1attribute-
The fingerprint of this downvote reason.
type string<members>1:::-
Container element for a list of members who’ve provided this downvote.
timestamp1attribute::::-
The timestamp of when the most recent member provided the downvote.
type ISO 8601 date and time<member>1+::::-
The member that provided this downvote.
type<member>
A.12. Access Permission Result Types
<grantedAccessPermissionMembers>-
The root element of the response for the list of members you have explicitly granted access to your entries. If empty, ALL members will be able to see your entries.
count1attribute-
The count of members who have been granted explicit access to your entries.
<members>1+-
A member who has been granted explicit access to your entries.
type<member>
<grantAccessPermissionResult>-
The root element of the response after granting a member access to your entries.
<member>1-
The member to whom you granted access to your entries.
type<member>
<revokeAccessPermissionResult>-
The root element of the response after revoking a member’s access to your entries.
<member>1-
The member from whom you revoked previously-granted access to your entries.
type<member>