Digital Measurement Content Asset Metadata: Difference between revisions

From Engineering Client Portal

No edit summary
 
(2 intermediate revisions by the same user not shown)
Line 45: Line 45:
|filetype
|filetype
|dcrassetmetadata  
|dcrassetmetadata  
|-
|-
|initid
|initid
Line 56: Line 54:


* [.gz] - non splittable, requires multiple files, if data larger than 32 MB
* [.gz] - non splittable, requires multiple files, if data larger than 32 MB
|-
|starttime
|start date and hour of the data in the file in UTC
* Example format: yyyymmddhh
|-
|endtime
|end date and hour (not inclusive)  of the data in the file in UTC
* Example format: yyyymmddhh
|}
|}


Line 71: Line 59:
Empty <code>'''_SUCCESS'''</code> file should be provided to indicate that data delivery for a particular hour and split is completed (even if there is no data for that particular hour and split).
Empty <code>'''_SUCCESS'''</code> file should be provided to indicate that data delivery for a particular hour and split is completed (even if there is no data for that particular hour and split).


=== Manifest File ===
=== Data delivery example for hour 10 (10:00:00 AM UTC to 10:59:59 PM UTC): ===
A manifest should be provided which contains metadata related to the uploaded file(s). The manifest is a text file in .json format that implements the AWS unload manifest file format. It has the same name as the data file, but has the <code>'''_manifest'''</code> suffix.
<code>useast1-nlsn-w-dig-sei-acme-feeds-prod/dcrassetmetadata/2022/06/12/10/
acme_dcrassetmetadata_a9ddf15ea054ea415718767ea6


Example:
useast1-nlsn-w-dig-sei-acme-feeds-prod/dcrassetmetadata/2022/06/12/10/_SUCCESS</code>
<syntaxhighlight lang="JSON">
{
   "entries": [
     {"url":"s3://bucket/prefix/0000_object_00.snappy.parquet", "mandatory":true},
     {"url":"s3://bucket/prefix/0001_object_00.snappy.parquet", "mandatory":true},
     {"url":"s3://bucket/prefix/0002_object_00.snappy.parquet", "mandatory":true}
     {"url":"s3://bucket/prefix/0003_object_00.snappy.parquet", "mandatory":true}
   ],
 
   "meta": {
     "schema_version": "S2SV1.7.0",
     "accreditation_status": "0",
     "start_time": "1710154800",
     "end_time": "1710158399",
     "record_count": 31337
   }
}
</syntaxhighlight>
 
The manifest contains a complete list of all files provided for a given period and split, as well as a meta data (a.k.a. header) which should include the following attributes:
{| class="wikitable"
!'''Parameter'''
!'''Description'''
!'''Required'''
!'''Specified'''
!'''Format / Example'''
!'''Type'''
|-
|schema_version
|Schema Version
|Yes
|Nielsen
|<code>S2SV1.7.0</code>
|String
|-
|accreditation_status
|Accreditation Status
|No
|Client
|<code>MRC = 1</code>
|String
|-
|data_start_time
|Data Start Time (min)
|Yes
|Client
|Format:32-bit unsigned int Unix time in seconds
Example: <code>1710154800</code>
|String
|-
|data_end_time
|Data Start Time (max)
|Yes
|Client
|Format:32-bit unsigned int Unix time in seconds
Example: <code>1710158399</code>
|String
|-
|record_count
|Number of records in data file
|Yes
|Client
|Example: <code>31337</code>
|Number
|}
 
=== Data delivery example for hour 11 (11:00:00 AM UTC to 11:59:59 PM UTC): ===
<code>useast1-nlsn-w-dig-sei-acme-feeds-prod/dar-exposure/ios/2022/05/25/11/acme_dtvr-exposure_a9ddf15ea054ea415718767ea6_P47C2495B-BBBA-56DE-BE99-14758F92F034_2024031111_2024031112_0000.json
 
useast1-nlsn-w-dig-sei-acme-feeds-prod/dar-exposure/ios/2022/05/25/11/acme_dtvr-exposure_a9ddf15ea054ea415718767ea6_P47C2495B-BBBA-56DE-BE99-14758F92F034_2024031111_2024031112_0001.json
 
useast1-nlsn-w-dig-sei-acme-feeds-prod/dar-exposure/ios/2022/05/25/11/acme_dtvr-exposure_a9ddf15ea054ea415718767ea6_P47C2495B-BBBA-56DE-BE99-14758F92F034_2024031111_2024031112_0002.json
 
useast1-nlsn-w-dig-sei-acme-feeds-prod/dar-exposure/ios/2022/05/25/11/acme_dtvr-exposure_a9ddf15ea054ea415718767ea6_P47C2495B-BBBA-56DE-BE99-14758F92F034_2024031111_2024031112_0003.json
 
useast1-nlsn-w-dig-sei-acme-feeds-prod/dar-exposure/ios/2022/05/25/11/acme_dtvr-exposure_a9ddf15ea054ea415718767ea6_P47C2495B-BBBA-56DE-BE99-14758F92F034_2024031111_2024031112_manifest
 
useast1-nlsn-w-dig-sei-acme-feeds-prod/dar-exposure/ios/2022/05/25/11/_SUCCESS</code>


== SLA ==
== SLA ==
The files must be delivered into the proper S3 bucket within 3 hours of the start of that hourly viewing file interval. For example, files from 1:00 AM to 2:00 AM must be delivered before 4 AM.
The files must be delivered into the proper S3 bucket within 2 hours of the broadcast. For example, files from 1:00 AM to 1:15 AM must be delivered before 3:00 AM. The metadata file should be delivered to an S3 bucket before any Viewing file has been delivered.  


== Schema Parameter Definitions ==
== Schema Parameter Definitions ==
Line 164: Line 75:
!'''Format / Example'''
!'''Format / Example'''
|-
|-
| colspan="2" |apn
| colspan="2" |assetid
|Application name
|In-house id used for a video asset (TMS ID if available)
|Yes
|Yes
|
|
Format: alphanumeric
Format: alphanumeric,


Example:
Example: VID123456789
<code>BestAppIOS</code>
|-
| colspan="2" |apv
|Application version
|Yes
|
Format: alphanumeric


Example:
AssetId needs to be unique at episode level
<code>21.5</code>
|-
|-
| colspan="2" |sessionid
| colspan="2" |assetname
|Unique, client generated value that represents the start of a user session. "Session" is defined as continuous (flexible) interaction with an application that may span multiple streams.
|In-house video asset name,
for troubleshooting
|Yes
|Yes
|
|
Format: alphanumeric
Format: alphanumeric


Example: Random GUID:
Examples: TMS ID, EIDR ID or Episode title
<code>cdcde33c-b62f-4f17-a9c8-0db4f78e09d6</code>
|-
| colspan="2" |streamid
|ID for every new instance of exposure to a different asset
|Yes, if no sessionid
|
Format: alphanumeric
 
Example: Random GUID:
<code>d7a909f1-5e77-4af7-8a9b-f2…</code>
|-
|-
| colspan="2" |streamended
| colspan="2" |programname
|Stream is known to have ended in this file
|Name of the program
|Yes
|Yes
|
|
Format: integer
Format: alphanumeric, Max 25 char’s; no special characters
 
Example:
*<code>1</code> (stream continues in subsequent file),
*<code>2</code> (stream closed)
|-
| colspan="2" |publisher_user_id
|Publisher-specific user ID (must remain persistent indefinitely)
|No
|
Format: alphanumeric
 
Example:
Salted, hashed user ID:
 
<code>8f434346648f6b96d9dda901c5…</code>
|-
| colspan="2" |provider_user_id
|Provider-specific user ID (must remain persistent indefinitely)
|No
|
Format: alphanumeric


Example: Salted, hashed user ID:
Example: Nightly News
 
<code>2cf24dba5fb0a30e26e83b2ab9…</code>
|-
|-
| colspan="2" |assetid
| colspan="2" |episodetitle
|In-house id used for a video asset (TMS ID if available)
|Title of the episode
|Yes
|Yes
|
|
Format: alphanumeric
Format: alphanumeric
Example:
<code>VID123456789</code>
|-
|-
| colspan="2" |<s>nielsen_id3_tag</s>
| colspan="2" |episodelen
|<s>Encrypted Nielsen ID3 Tag. Contains SID (Source Identifier) codes (PC - Program Content  &  FD - Final Distributor).</s>
|Total episode length
|<s>Optional</s>
|
<s>Format: alphanumeric;</s>
 
<s>Example:</s>
 
<code><s>www.nielsen.com/X100zdCIGeIlgZnkYj6UvQ==/X100zdCIGeIlgZnkYj6UvQ==<wbr />/AAAB2Jz2_k74GXSzx4npHuI<wbr />_JwJd3QSUpW30rDkGTcbHEzIMWleCzM-uvNOP9fzJcQMWQLJqzXMCAxParOb5sGijSV9dNM3QiBniJYGZ5GI<wbr />-lL1fXTTN0IgZ4iWBmeRiPpS9AAAAAAAAAAAAAAAAAAAAAFJWFM5SVhTONNU=<wbr />/00000/00000/00</s></code>
 
<s>Note: Only Data Tags should be included, INFO Tags should not.</s>
 
<s>INFO Tag is characterized by the following CID prefix: X100zdCIGeIlgZnkYj6UvQ==</s>
 
<s>Example INFO Tag (not desired):</s>
 
<code><s>www.nielsen.com/X100zdCIGeIlgZnkYj6UvQ==/X100zdCIGeIlgZnkYj6UvQ==/<wbr />AAICoyitYqlxT7n6aZ0oMCGheFi4CXFp46AMUPZz1lMr_M9tr3<wbr />_cjee1SHqxrOiVerMDLeyn9xzocZSKwi746Re8vNOtpNCAZjYABs_<wbr />J0R25IHpvOc1HS8QHGgD5TgOJeS6gX100zdCIGeIlgZnkYj6UvVJWFNhSVhTiPE0=<wbr />/00000/46016/00</s></code>
|-
| colspan="2" |<s>gracenote_id</s>
|<s>Gracenote TMS ID (If available) should be passed for all telecasted content.</s>
|<s>Required if id3 not provided</s>
|
<s>Format: 14 character string. Normally leading with 2 alpha characters ('EP', 'MV', 'SH' or 'SP'), followed by 12 numbers.</s>
 
<s>Example:
<code>SH009311820022</code></s>
|-
| colspan="2" |<s>station_id</s>
|<s>GraceNote station ID that identifies the station</s>
|<s>Required if id3 not provided</s>
|
<s>Format: alphanumeric</s>
|-
| colspan="2" |<s>program_name</s>
|<s>Name of program</s>
|<s>Required if id3 not provided</s>
|
<s>Format: alphanumeric, Max 25 characters; no special characters.</s>
 
<s>Example:
<code>Nightly News</code></s>
 
<s>See below for more specific guidance on Program Name.</s>
|-
| colspan="2" |<s>network_affiliate</s>
|<s>Network affiliation of a station</s>
|<s>Required if id3 not provided</s>
|
<s>Format: alphanumeric</s>
 
<s>Example:
<code>xyz</code></s>
|-
| colspan="2" |<s>channel_id</s>
|<s>ID of channel</s>
|<s>Required if id3 not provided</s>
|
<s>Format: integer</s>
|-
| colspan="2" |<s>channel_name</s>
|<s>Name of channel</s>
|<s>Required if id3 not provided</s>
|
<s>Format: alphanumeric</s>
|-
| colspan="2" |<s>callsign</s>
| <s>FCC assigned unique identifier for a transmitter station</s>
|<s>Required if id3 not provided</s>
|
<s>Format: alphanumeric</s>
|-
| colspan="2" |dma
|Designated Market Area where viewing occurred
|Yes
|<code>501</code>
|-
| colspan="2" |<s>ad_load_flag</s>
|<s>linear or dynamic ad load</s>
|<s>Yes</s>
|
<s>Format: integer;</s>
 
*<s><code>0</code> = Default / Unknown</s>
*<s><code>1</code> = linear ad load</s>
*<s><code>2</code> = dynamic ad load</s>
|-
| colspan="2" |ad_support_flag
|Intended method of monetizing the content
|Yes
|Yes
|
|
Format: integer;
Format: Integer (seconds)
 
*<code>0</code> = (no ads),
*<code>1</code> = (content is supported by ads)
*<code>2</code> = (not known)


Note: set to "<code>1</code>" if content ad support was intended but did not occur
Example: 3600
|-
| colspan="2" |position
|Array of contiguous content viewing.


For viewing gaps < than 1 second, the gap can be smoothed over
For continuous live stream or unknown duration use 24*60*60 (86400)


See below for additional details on position array parameters
Can be updated with a replacement asset file when the live playout ends, and the exact duration becomes known
|Yes
|
Format:
 
<syntaxhighlight lang="JSON">
"position": [{
 "referencestart":"[timestamp]",
 "referenceend":"[timestamp]"
 "playheadstart":"[playhead position]",
 "playheadend":"[playhead position]"
}]
</syntaxhighlight>
|-
|-
|{
| colspan="2" |isfullepisode
|referencestart
|Full episode Flag
|Wall clock reference start time
(indicates full episode or a clip)
|Yes
|Yes
|
|
Format: Unix timestamp in 32-bit unsigned int in seconds
Format: integer – 0 (not full episode), 1 (full episode)
 
Example:
<code>1577858505</code>
|-
|-
| colspan="2" |genre
|Nielsen genre code
|No
|
|
|referenceend
Format: alphanumeric
|Wall clock reference end time
|Yes
|
Format: Unix timestamp in 32-bit unsigned int in seconds


Example:  
Example: CV (comedy variety), N (news)
<code>1577858775</code>
|-
|-
|
| colspan="2" |airdate
|playheadstart
|The date the episode was first made available on Traditional TV
|Content position start time
|Yes
|
Format: Unix timestamp in 32-bit unsigned int in seconds or integer indexed from 0 (typical for VOD)
 
Example:
<code>1577858515</code>
|-
|<nowiki>}</nowiki>
|playheadend
|Content position end time
|Yes
|
Format: Unix timestamp in 32-bit unsigned int in seconds or integer indexed from 0 (typical for VOD)
 
Example:
<code>1577858785</code>
|-
| colspan="2" |viewedads
|Array of ads viewed by client.
 
See below for additional details on position array parameters
|Yes
|
Format:
<syntaxhighlight lang="JSON">
"viewedads":
[
  {
      "adpod":"0",
      "adposition":"0",
      "adid":"0",
      "isdarid":"0",
      "adstart":"utc",
      "adend":"utc",
      "advisible":"0",
      "adfocus":"0",
      "adaudio":"0"
  }
]
</syntaxhighlight>
|-
|{
|adpod
|AdPod sequence number.
 
Increment for each AdPod. If the same AdPod is played out twice (due to rewind), still increment the AdPod sequence to reflect the sequence the AdPods are played.
|Yes
|
Format: Integer
 
Example:
<code>2</code>
|-
|
|adposition
|Ad sequence number.
 
Increment each time a new Ad is encountered. If the same Ad is played out twice (due to rewind), still increment the Ad sequence to reflect the sequence the Ads are played
|Yes
|
Format: Integer
 
Example:
<code>1</code>
|-
|
|adid
| The unique identifier for this Ad
|No
|No
|
|
Format: alphanumeric
Format: US ET:  YYYYMMDDHHMMSS


In-house AdId, Industry AdID, AdNetwork ID or DAR placement ID
Example: 19970716231256
|-
|
|isdarid
|Is the AdId being passed a Nielsen DAR placement ID
|Required if AdID
|
Format: integer
*<code>0</code> = is NOT DAR ID;
*<code>1</code> = is a DAR ID
|-
|
|adstart
|Wall clock reference start time
|Yes
|
Format: Unix timestamp in 32-bit unsigned int in seconds
|-
|<nowiki>}</nowiki>
|adend
|Wall clock reference end time
|Yes
|
Format: Unix timestamp in 32-bit unsigned int in seconds
|-
| colspan="2" |device_id
|Mobile Ad ID (IDFA, ADID), Connected Device ID
|Yes,if


available
For the US, the time should be specified as Eastern Timezone. Other countries will use a local timezone.
|<code>A487421B-XXXX-YYYY-8343-E3BBB66E44F2</code>
|-
| colspan="2" |hem_sha256
|SHA-256 hashed email


Note: email normalization rules applied before hashing
This value can be updated with a replacement asset file if the original air date changes
|Strongly Preferred
|<code>55C06A30DAA5D5F382FDEB8C702EC57875CC9D91A7C78BB620053FD81DC…</code>
|-
| colspan="2" |luid
|Living Unit ID - Experian Household ID
|Yes, for CTV if available
|<code>B0EOFEDgD</code>
|-
|-
| colspan="2" |uoo
| colspan="2" |webpageurl
|User opt out flag for demographic measurement
|Website Identifier, for diagnostics, can be turned off/on as needed
|Yes
|Optional
|
|
Format: integer
Format: alphanumeric  (max 2k)
*<code>0</code> = not opt-out
*<code>1</code> = opt-out
|-
| colspan="2" |xdua
|Device HTTP User Agent string
|Yes
|
Format: alphanumeric,


Example:  
Example:<nowiki>https://www.myapp.com/video.html</nowiki>
<code>Apple-iPhone1C2/801.293</code>
|-
| colspan="2" |xff
|IP address
|Yes
|
Format: xxx.xxx.xxx.xxx
|-
| colspan="2" |psudo_id_sha256
|Hashed Device User Agent string + IP address
|No
|<code>421c76d77563afa1914846b010bd164f395bd34c2102e5e99e0cb9cf173…</code>
|-
| colspan="5" | The following 4 parameters become mandatory if Device User Agent String (UAS) is not available
|-
| colspan="2" |device_platform
|Device platform(mobile, desktop, connected device)
|Required if no UAS
|<code>DSK</code>, <code>MBL</code>, <code>OTT</code>
|-
| colspan="2" |device_type
|Device type for connected devices
|Required if no UAS
|<code>AMN</code>, <code>APL</code>, <code>DVD</code>, <code>GGL</code>, <code>PSX</code>, <code>RKU</code>, <code>STB</code>, <code>STV</code>, <code>XBX</code>
|-
| colspan="2" |os_group
|Operating system of mobile devices. All other device should be NA
|Required if no UAS
|<code>IOS</code>, <code>DROID</code>, <code>NA</code>
|-
| colspan="2" |device_group
|Type of device: phone, tablet, desktop, set top box (CTV or OTT), unknown
|Required if no UAS
|<code>DSK</code>, <code>PHN</code>, <code>TAB</code>, <code>STV</code>, <code>DVD</code>, <code>PMP</code>, <code>OTHER</code>, <code>STB</code>, <code>XBX</code>, <code>PSX</code>, <code>AMN</code>, <code>APL</code>, <code>GGL</code>, <code>RKU</code>, <code>UNWN</code>
|-
| colspan="5" | The following 4 parameters become mandatory if IP Address is not available
|-
| colspan="2" |robotic_flag
|Flag indicating IAB bot rule if IP address present in IAB bot list
|Required if no IP
|Format: integer
*<code>0</code> = not bot,
*<code>1</code> = bot
|-
| colspan="2" |zip_code
| ZIP code where viewing occurred
|Required if no IP
|<code>10001</code>
|-
| colspan="2" |country
|Country ISO 3166 ALPHA-2
|Required if no IP
| <code>US</code>, <code> CA</code>, etc.
|}
|}
Note: All parameters are case sensitive.
Note: All parameters are case sensitive.

Latest revision as of 12:34, 30 October 2025

Engineering Portal / Digital / US DCR & DTVR / Digital Measurement Content Asset Metadata

The Nielsen "Server-To-Server" method of collecting DCR data requires the distributor to make available each Asset (Episode) referenced in the DCR Viewing file in a separate metadata file (a.k.a metadata catalog). In the case of a stream that has one version with blackouts and one without, one with mobile and one non-mobile, each stream would have its own unique asset id.

Delivery Specifications

Consolidated list of asset metadata catalog that is up to date for a particular hour needs to be provided in a single file following the naming convention specified below. The metadata needs to be in a json lines format, including a header line. A reasonable effort needs to be made to remove expired/inactive asset ids, to avoid size creep.

S3 Bucket and Prefix Naming Convention

useast1-nlsn-w-dig-sei-<partnerid>-feeds-<env>/<filetype>/yyyy/mm/dd/hh/<object>

Name Description
partnerid Abbreviation provided by Nielsen for each provider or publisher
env test or prod
filetype dcrassetmetadata


split a separate data split, can be by platform (ex.: ios, browser, android, ctv), by country (us, ca, jp, etc.), by publisher, by team, etc. or “all” (if data is provided in one split).
yyyy/mm/dd/hh
  • yyyy - year
  • mm - month
  • dd - date padded with 0 example 01, 02,..., 31
  • hh - hour padded with 0 example 00, 02,..., 23

S3 Bucket and Prefix Naming Convention

<partner_id>_<file_type>_<int_id>[.gz]

Name Description
partnerid Abbreviation provided by Nielsen for each provider or publisher
filetype dcrassetmetadata
initid integration id: unique identifier provided by Nielsen
[.gz] Uncompressed json lines (no extension) -  splittable, ideal for medium size files, less than 5 GB, preferred for metadata.

Optional compression formats supported:

  • [.gz] - non splittable, requires multiple files, if data larger than 32 MB

Success File

Empty _SUCCESS file should be provided to indicate that data delivery for a particular hour and split is completed (even if there is no data for that particular hour and split).

Data delivery example for hour 10 (10:00:00 AM UTC to 10:59:59 PM UTC):

useast1-nlsn-w-dig-sei-acme-feeds-prod/dcrassetmetadata/2022/06/12/10/ acme_dcrassetmetadata_a9ddf15ea054ea415718767ea6

useast1-nlsn-w-dig-sei-acme-feeds-prod/dcrassetmetadata/2022/06/12/10/_SUCCESS

SLA

The files must be delivered into the proper S3 bucket within 2 hours of the broadcast. For example, files from 1:00 AM to 1:15 AM must be delivered before 3:00 AM. The metadata file should be delivered to an S3 bucket before any Viewing file has been delivered.

Schema Parameter Definitions

Parameter Description Required Format / Example
assetid In-house id used for a video asset (TMS ID if available) Yes

Format: alphanumeric,

Example: VID123456789

AssetId needs to be unique at episode level

assetname In-house video asset name,

for troubleshooting

Yes

Format: alphanumeric

Examples: TMS ID, EIDR ID or Episode title

programname Name of the program Yes

Format: alphanumeric, Max 25 char’s; no special characters

Example: Nightly News

episodetitle Title of the episode Yes

Format: alphanumeric

episodelen Total episode length Yes

Format: Integer (seconds)

Example: 3600

For continuous live stream or unknown duration use 24*60*60 (86400)

Can be updated with a replacement asset file when the live playout ends, and the exact duration becomes known

isfullepisode Full episode Flag

(indicates full episode or a clip)

Yes

Format: integer – 0 (not full episode), 1 (full episode)

genre Nielsen genre code No

Format: alphanumeric

Example: CV (comedy variety), N (news)

airdate The date the episode was first made available on Traditional TV No

Format: US ET:  YYYYMMDDHHMMSS

Example: 19970716231256

For the US, the time should be specified as Eastern Timezone. Other countries will use a local timezone.

This value can be updated with a replacement asset file if the original air date changes

webpageurl Website Identifier, for diagnostics, can be turned off/on as needed Optional

Format: alphanumeric  (max 2k)

Example:https://www.myapp.com/video.html

Note: All parameters are case sensitive.

Retrieved from "https://nielsentest.mywikis.net/w/index.php?title=Digital_Measurement_Content_Asset_Metadata&oldid=6932"