Arik L.

So I would assign values for the z_id, z_cluster, z_zsource etc.. Like this for example?

data = {
    'z_zid':[4596871, 4596872, 4596873],
    'z_prediction': [0, 0 ,0],
    'z_score': [0, 0, 0,],
    'z_isMatch': [0 ,0 ,0],
    'z_cluster': ["1737559111670:9", "1737559111670:9", "1737559111670:9"],
    'name': ['Digital Workplace Results (DWR)', 'Digital Marketing Solutions', 'Digital Webs'],
    'postcode': [None, None, '3008'],
    'domain': ['dwresults.com', 'digitalmarketingsolutions.com.au', 'mydigitalwebs.com.au'],
    'country': ['au', 'au', 'au'],
    'registration_number': [None, None, None],
    'location': ['Melbourne', 'Melbourne', 'Melbourne Melbourne, VIC 3008'],
    'jurisdiction_code': ['au', 'au', 'au'],
    'latitude': [None, -37.8142176, -37.8142176],
    'longitude': [None, 144.9631608, 144.9631608],
    'source': ['linkedin', 'glass', 'glass'],
    'source_id': ['2668211_au', '4615a978-8999-4321-b984-231918a132eb', 'b639ee8f-0699-41c1-bd4f-97df9a35de0b'],
    'FirmId': ['4acb4472-bffc-429a-9745-a75437408e22', 'fd2a4177-81d1-49ec-8878-463c959ddd6b', '66d4a8a6-0e78-4f88-91f1-20497d5af8b0'],
    'z_zsource':['er_test', 'er_test', 'er_test']
}

Is it possible to do this in databricks with the python sdk? e.g. adding a parquet file to the MARKED_DIR?

Hi Sonal G. just following what you said about sending a copy of matches or I assume unmatched records to zingg directly, rather than going through the manual labelling process, are there any docs you can send me outlining how to do this?

Something I've noticed today while doing labelling the matches is that one of the three sources (the biggest one) hasn't shown up at all during the labelling stage. I'm wondering as to the reason for that? And if that is something that can be fixed with either sample size or order of blocking strategies?

So is it better to have 1 main pipe rather than three separate ones in this case? Or there's no difference? No the 800k are not the same records but they are records that have been matched previously with our existing matching logic. So we are fairly confident that they are matches. Some will have matched on e.g. name and domain or exact registration number. And just on the lat, long question, do you know of a way to accommodate for this, other than writing custom code? Thanks!

Currently, I have three data sources of firmographic data for this POC. If this approach shows promise, we plan to add more data sources in the future. For now, I’ve concatenated all three data sources into a single file with a unified schema, resulting in approximately 6 million records (only for Australia). I currently have one input pipe, but modifying the load step to accommodate three separate pipes wouldn’t be an issue. These are the current match fields and strategies I’m aiming for:

source_id = FieldDefinition("source_id", "string", MatchType.DONT_USE)
name = FieldDefinition("name", "string", MatchType.FUZZY)
domain = FieldDefinition("domain", "string", MatchType.EXACT)
country = FieldDefinition("country", "string", MatchType.DONT_USE)
jurisdiction_code = FieldDefinition("jurisdiction_code", "string", MatchType.DONT_USE)
location = FieldDefinition("location", "string", MatchType.FUZZY)
postcode = FieldDefinition("postcode", "string", MatchType.FUZZY)
latitude = FieldDefinition("latitude", "double", MatchType.DONT_USE)
longitude = FieldDefinition("longitude", "double", MatchType.DONT_USE)
source = FieldDefinition("source", "string", MatchType.DONT_USE)
registration_number = FieldDefinition("registration_number", "string", MatchType.EXACT)
FirmId = FieldDefinition("FirmId", "string", MatchType.DONT_USE)

# Put these in order from most important to least important!  Zingg prioritizes them in that order.
field_defs = [registration_number
            ,name
            ,domain
            ,country
            ,postcode
            ,location
            ,jurisdiction_code
            ,latitude
            ,longitude
            ,source
            ,source_id
            ,FirmId]

To prioritise fields, I’ve ordered them from most important to least important based on our current matching rules, although perhaps having e.g. exact registration number as the first field wouldn't be the best blocking strategy. I’m not currently using latitude and longitude as I don’t believe the numeric MatchType is a good fit. However, I’m open to suggestions if there’s a better way to leverage these fields effectively. I was also curious if "balancing the data" by including some known matches—for example, reducing the 6 million records to 2 million, where approximately 800,000 are known matches—would be an effective strategy?

Thanks Sonal G. for your help! I had another quick question, If I use different pipes for different data sources rather than join them into one file, will I be unable to link records from within a single data source? Does Zingg assume no entities need to be resolved within a single pipe? What is the advantage of doing one or the other?

Working now thanks!

Sonal G. thanks for the quick reply! I think I may have missed this step, all I've done is add zingg==0.4.0 to the cluster library