Skip to main content

Data Warehouse Event Sync Issues in Vitally

Issues with syncing event data from Snowflake to Vitally due to timestamp filtering, missing required fields, or duplicate account ID columns.

Written by Ethan Patrick

Problem Description:

Users may encounter issues when pulling event data from a data warehouse (Snowflake, BigQuery, Redshift, etc.) into Vitally. While the preview query runs successfully, events may not appear in Vitally after the source is enabled and saved.

Potential Reasons for the Issue:

1. Incremental Sync Constraints:

  • If the sync has never run before, events will not appear until the first scheduled sync completes. Syncs run on a schedule (up to 24 hours) and do not trigger immediately on save.

2. Incorrect Filtering Conditions:

  • The column compared against $LAST_SYNC_TIMESTAMP reflects when the event occurred rather than when it was loaded into the warehouse. If data is loaded in batches, events loaded after the last sync can be missed even if they happened before it.

  • Note: $LAST_SYNC_TIMESTAMP is required for all event source queries. The platform will reject any query that omits it.

3. Missing Required Fields:

  • Custom objects require specific columns:

    • timestamp for event occurrence time.

    • object_id as a unique identifier.

    • external_account_ids or external_organization_ids to associate with an account or organization.

    • name to label the object.

  • If any of these fields are empty or unmapped, Vitally may fail to sync them.

4. Unexpected Sync Errors:

  • Errors displayed in the UI could indicate that required fields are not properly mapped under Accounts, Users, Events, or Custom Objects.

5. Duplicate Account ID Columns:

  • Having multiple account ID columns in the dataset may cause sync failures.

6. Misconfigured Integration Settings:

  • Incorrect integration IDs, such as an invalid Salesforce ID, may prevent events from syncing correctly.

Step-by-Step Solutions:

1. Verify Timestamp Conditions

  • Confirm your WHERE clause compares $LAST_SYNC_TIMESTAMP against a column that reflects when data was loaded into the warehouse (e.g., inserted_at, loaded_at), not when the event occurred. Using an event-occurrence timestamp can cause batch-loaded data to be missed.

  • If events are not appearing after the first save, confirm the sync has had time to run on its scheduled interval. The first ingestion does not occur immediately on save.

  • Note: $LAST_SYNC_TIMESTAMP is required and cannot be removed from event source queries. Any query that omits it will be rejected on save.

2. Review Required Fields for Custom Objects

  • Ensure the following fields are included in the query:

    • timestamp

    • object_id

    • external_account_ids or external_organization_ids

    • name

  • Map all required fields in the Vitally interface.

3. Investigate UI Errors

  • If an error appears at the top of the screen, check if any required fields remain unmapped in:

    • Accounts

    • Users

    • Events

    • Custom Objects

  • Ensure no empty fields exist in these categories.

4. Check for Duplicate Columns

  • Look for duplicate Account ID columns in the dataset.

  • Ensure only one external_account_ids field is present.

5. Re-run the Sync

  • After making necessary adjustments, run the sync again.

  • Verify if events and custom objects now appear in Vitally.

Correcting IDs and Resyncing Events

If you suspect that a Salesforce ID for the integration was configured incorrectly, follow these steps to fix the issue:

  1. Copy the query for the event type you want to resync.

  2. Remove the event from the integration in Vitally.

  3. Update the integration with the corrected ID.

  4. Refresh your browser to ensure updates are reflected.

  5. Recreate the event source for the affected event type.

  6. Paste the previously copied query back into the event source settings.

  7. Save and update the integration.

Additional Tips:

  • Reference Vitally Documentation:

  • Syncing Events Guide

  • Check Sync Logs:

  • Use Vitally's sync logs to determine if records were processed successfully.

  • Test with a Smaller Dataset:

  • Run queries with a limited dataset to isolate potential issues.

  • Ensure Correct API Call Usage:

  • Use a Track Call instead of a Group Call to accurately log and sync events. Group Calls are used only for updating trait data, not event information.

  • Always include the full external ID for the entity (e.g., organization or domain) when making a Track Call.

FAQs:

How do I resolve issues caused by incorrect Salesforce IDs?

Reconfigure the Salesforce ID and follow the steps to resync events, ensuring all affected sources are updated.

Why are my events not syncing correctly despite updating integration traits?
Ensure you are using a Track Call instead of a Group Call for event syncing and include the full external ID in the call.

My data is not appearing in Vitally. What should I check first?
Confirm your WHERE clause compares $LAST_SYNC_TIMESTAMP against a warehouse load timestamp rather than an event-occurrence timestamp. If syncing for the first time, confirm the scheduled sync has had time to run — events do not appear immediately after saving. Note that $LAST_SYNC_TIMESTAMP is required and cannot be removed from event source queries.

Why are some events missing in Vitally?
Check if all required fields (timestamp, object_id, external_account_ids, name) are present and mapped correctly.


​What does the sync error at the top of the screen mean?

It likely indicates missing required fields. Review field mappings in Accounts, Users, Events, or Custom Objects.

My custom object sync is failing. What could be the issue?
Ensure that all required fields are correctly defined and that there are no duplicate Account ID columns.

How can I confirm if my sync worked?
Check Vitally's sync logs to see if events and custom objects were successfully imported.

Related Articles
Salesforce
S3
Syncing Custom Objects
Syncing NPS Responses
Syncing Events
Did this answer your question?