Fetch new Social Contacts

Version 1
    This document was generated from CDN thread

    Created by: Tod Famous on 15-06-2011 04:57:20 PM
    p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 13.0px Consolas}
    From a partner:
    We are periodically fetching campaign results and processing all
    articles.  We need to figure out how to fetch only those articles we
    haven't seen before.  The campaign results api doesn't appear to have a
    way to fetch articles newer than a specified timestamp or postId (only
    older).  To get the unseen articles, it appears we have to fetch from
    the top of the results feed and work backwards until we're sure we've
    read them all.  This becomes hard when a campaign is configured with
    multiple feeds, each having their own polling interval.  If we had a
    feed configured to poll hourly we'd always have to fetch from the "top"
    of the results and work back at least an hour to ensure we got all the
    unseen articles.

    Subject: RE: Fetch new Social Contacts
    Replied by: Tod Famous on 15-06-2011 05:01:07 PM
    We suggest you use the countSince API to find out how many new posts there are and then use the "numberOfResults" query param on campaign results to only return that many.

    An alternative would be to use the HTTP post notification method to get asynchronously notified with each new social contact.

    This would work in 8.5(2) whereby each time a tag was applied, you would get notified.  Tag application would need to be "manual".  However in 8.5(3) which will be released in a week or so we will support auto-tagging so that you could configure all social contacts to be tagged and therefore all social contacts to generate a "notification".

    8.5(3) documentation has just been posted.  API doc and software will be coming in about a week.
    http://docwiki.cisco.com/wiki/SocialMiner

    Subject: RE: Fetch new Social Contacts
    Replied by: Brett Bandy on 16-06-2011 05:46:22 PM
    We suggest you use the countSince API to find out how many new posts there are and then use the "numberOfResults" query param on campaign results to only return that many.

     
    When I use the countSince API how can I be sure:
     
    a) that no new posts have arrived since I executed the countSinceAPI?
    b) that the 1st X post are the new ones?  If I have a campaign with a feed that's fetched every minute and a feed that's fetched every hour, it would be possible that the new posts are from the feed fetched every hour and may not be the 1st X posts fetched.
     
    thanks,
    Brett

    Subject: RE: Fetch new Social Contacts
    Replied by: Tod Famous on 19-06-2011 03:55:24 PM
    a) Yes, there would be a small window.  Perhaps get more then you need and de-dup.

    b) asking development for their view on this question.


    Tod
    (Note: we realize this isn't optimal.  Hence the http may be better route to go long term or the script filter we're working on in 8.5(4) may be best.  We're working on 8.5(4) now.  We should have something on our hosted system in July.)

    Subject: RE: Fetch new Social Contacts
    Replied by: John Russell on 20-06-2011 09:39:23 AM

     
    When I use the countSince API how can I be sure:
    ...
    b) that the 1st X post are the new ones?  If I have a campaign with a feed that's fetched every minute and a feed that's fetched every hour, it would be possible that the new posts are from the feed fetched every hour and may not be the 1st X posts fetched.
     

    If you are processing all of the SC's in the campaign, then a simpler way to check for new ones is to set the status of each SC to handled when you do something to them.  Then you can get the campaign results only for UNREAD and/or RESERVED and then handle all of those.  Then there are no gaps and you won't need to use two API calls each time.
     
    Does that work for your use case?

    Subject: RE: Fetch new Social Contacts
    Replied by: Brett Bandy on 20-06-2011 12:40:01 PM

    If you are processing all of the SC's in the campaign, then a simpler way to check for new ones is to set the status of each SC to handled when you do something to them.  Then you can get the campaign results only for UNREAD and/or RESERVED and then handle all of those.  Then there are no gaps and you won't need to use two API calls each time.
     
    Does that work for your use case?

     
    Unfortunately, its not bullet proof.  Is perfectly legal to set the contact state back from RESERVED to UNREAD, so I don't think I can rely on the contact's state to determine which posts are new. I'd still have to figure out some way to determine the difference between new, unseen posts and these posts that have had the state set back to UNREAD.
     
    I did notice that all new posts have a state of UNREAD and a null ccp:scstatususerid element.  Could I rely on this to indicate a new, unread post?  Is there some other way to get the contact's state set back to UNREAD, but not have a valid userId value?
     
    thanks,
    Brett

    Subject: RE: Fetch new Social Contacts
    Replied by: John Russell on 20-06-2011 03:03:20 PM
    That would probably work, since even if someone set it back to UNREAD their userid would be recorded as the user that did it.

    Subject: RE: Fetch new Social Contacts
    Replied by: Tod Famous on 20-06-2011 03:38:05 PM
    Note: this has a reporting implication.

    the handled time in SocialMiner is the time between the first time the Social Contact gets "reserved" until the social contact is handled.

    If the post goes reserved, then unread, then reserved, then handled.  We take the time from the first reserved.

    So if the external application reserves items, then it will dramatically increase the AHT.

    Tagging is probably a better approach to marking items.  For example, tag every item you've pulled with a tag that indicates it's been read.


    I'm not entirely sure what you're trying to do end to end with your integration.  We should probably review that holistically.

    I *think* what you are trying to do is look at the SC's and tag them or perhaps discard them.  If that's the case, then the SC's that have no tags are the SC's that you have yet to /process/.  So what you might do is  pull a bunch of unread SC's, process them and put tags on all of them (even if you are putting a tag that says "we aren't sure how to tag".  You can also delete the ones that you thing should be deleted etc.  Then you go pull a another set of SC's.  You would discard any that had been tagged as they have already been processed.

    Subject: RE: Fetch new Social Contacts
    Replied by: Brett Bandy on 20-06-2011 07:19:28 PM
    That would probably work, since even if someone set it back to UNREAD their userid would be recorded as the user that did it.

     
    Could I also use the postId to find unread posts?  The postId appears to be an increasing hex value.   From my testing it seems that newly processed (fetched by SocialMiner) articles are given a larger postId than previously fetched posts.  This holds true for articles, with an older published date, that were fetched from a feed with a large polling interval. 
     
    The countsince API uses the postID to determine the count since a last fetch, so it must provide some way to order articles regardless of article's published date.
     
    thanks,
    Brett

    Subject: RE: Fetch new Social Contacts
    Replied by: Brett Bandy on 20-06-2011 09:00:48 PM
    Unfortunately postId goes the wrong way.  It is used for paging backwards, so if you give it a postId it will return the results earlier than that postId.  We'd have to change the api to add an option to make it show you everything _after_ that postId.

     
    Sorry, I wasn't very clear w/ my question.   Let me try again.
     
    My campaign has feeds that have different polling intervals.  Posts from each feed would get added to the campaign at different times, but the postId would always be increasing.  If the countsince API returned 10 for a given postId, I would need to hunt around to find those 10 posts since they aren't guaranteed to be at the top of the campaign results feed.  What I was asking was if I could start at the top and work my way back using the postId until I found the 10 new posts?  If I compared each post's postId to my saved postId all "new" articles should have a larger postId.  
     
    As I mentioned, my testing seems to indicate that the postId is always increasing.  Can I rely on this to always be true?
     
    thanks,
    Brett Bandy
     

    Subject: RE: Fetch new Social Contacts
    Replied by: John Russell on 20-06-2011 08:39:43 PM

    Could I also use the postId to find unread posts?  The postId appears to be an increasing hex value.   From my testing it seems that newly processed (fetched by SocialMiner) articles are given a larger postId than previously fetched posts.  This holds true for articles, with an older published date, that were fetched from a feed with a large polling interval. 
     
    The countsince API uses the postID to determine the count since a last fetch, so it must provide some way to order articles regardless of article's published date.
     

     
    Unfortunately postId goes the wrong way.  It is used for paging backwards, so if you give it a postId it will return the results earlier than that postId.  We'd have to change the api to add an option to make it show you everything _after_ that postId.

    Subject: RE: Fetch new Social Contacts
    Replied by: John Russell on 21-06-2011 08:44:34 AM

    Sorry, I wasn't very clear w/ my question.   Let me try again.
     
    My campaign has feeds that have different polling intervals.  Posts from each feed would get added to the campaign at different times, but the postId would always be increasing.  If the countsince API returned 10 for a given postId, I would need to hunt around to find those 10 posts since they aren't guaranteed to be at the top of the campaign results feed.  What I was asking was if I could start at the top and work my way back using the postId until I found the 10 new posts?  If I compared each post's postId to my saved postId all "new" articles should have a larger postId.  
     
    As I mentioned, my testing seems to indicate that the postId is always increasing.  Can I rely on this to always be true?
     

     
    Ah I see.  The postId of the SC is a guid that we generate internally.  Its possible that the algorithm will always produce a larger hex value but it is absolutely not part of the API and not guaranteed to be the case all the time.  It could flip over at any time resulting in really weird bugs on your end. 
     
    I think your best bet right now might be the count since API and then fetching more than the result to handle any overlap and looking for duplicates.  Its very unlikely to give you problems and should work until you can bug Tod enough to put in a story for the actual feature which does sound very useful.