twitter

How to load a list of a user’s twitter friends/followers in your Windows Phone app

TwitterFriendsList_WP_Blog During the last days, I was adding a new function to UniShare that suggests users the Twitter handles while typing in some text. The base of this new function is a list of the user’s twitter friends. In this post, I am going to show you how to generate this list and save it for further use. First thing I need to talk about: you can only load a maximum of 3000 friends at once due to the rate limit on Twitter’s part. At the end of the post, I’ll give you also an idea how to continue this after the first 3000 users were loaded. For the most scenarios, 3000 should be enough to cover the major part of your users. To be able to load all friends we first create a Task that fetches 200 entries of our user’s friends list. To be able to perform paging through all entries, Twitter uses so called cursors, which are 64-bit signed integers (long). You can read more about that here on Twitter’s API documentation. Let’s have a look on this Task:

        private async Task<string> TwitterFriendsListString(long twitterCursor)
        {
            ShowProgressIndicator("generating mention suggestion base...");

            //needed for generating the Signature
            string twitterRequestUrl = "https://api.twitter.com/1.1/friends/list.json";

            //used for sending the request to the API
            //for usage of the url parameters, go to https://dev.twitter.com/docs/api/1.1/get/friends/list
            string twitterUrl = string.Format("https://api.twitter.com/1.1/friends/list.json?cursor={0}&screen_name={1}&count=200&skip_status=true&include_user_entities=false", twitterCursor, App.SettingsStore.twitterName);

            string timeStamp = GetTimeStamp();
            string nonce = GetNonce();

            string response = string.Empty;

            //we need to include all url parameters in the signature
            //IMPORTANT: Twitter's API demands them to be sorted alphabetically, otherwise the Signature will not be accepted
            string sigBaseStringParams = "count=200";
            sigBaseStringParams += "&" + "cursor=" + twitterCursor;
            sigBaseStringParams += "&" + "include_user_entities=false";
            igBaseStringParams += "&" + "oauth_consumer_key=" + twitterConsumerKey;
            sigBaseStringParams += "&" + "oauth_nonce=" + nonce;
            sigBaseStringParams += "&" + "oauth_signature_method=HMAC-SHA1";
            sigBaseStringParams += "&" + "oauth_timestamp=" + timeStamp;
            sigBaseStringParams += "&" + "oauth_token=" + App.SettingsStore.twitteroAuthToken;
            sigBaseStringParams += "&" + "oauth_version=1.0";
            sigBaseStringParams += "&" + "screen_name=" + App.SettingsStore.twitterName;
            sigBaseStringParams += "&" + "skip_status=true";
            string sigBaseString = "GET&";
            sigBaseString += Uri.EscapeDataString(twitterRequestUrl) + "&" + Uri.EscapeDataString(sigBaseStringParams);

            string signature = GetSignature(sigBaseString, twitterConsumerSecret, App.SettingsStore.twitteroAuthTokenSecret);

            string authorizationHeaderParams =
                "oauth_consumer_key=\"" + twitterConsumerKey +
                "\", oauth_nonce=\"" + nonce +
                "\", oauth_signature=\"" + Uri.EscapeDataString(signature) +
                "\", oauth_signature_method=\"HMAC-SHA1\", oauth_timestamp=\"" + timeStamp +
                "\", oauth_token=\"" + Uri.EscapeDataString(App.SettingsStore.twitteroAuthToken) +
                "\", oauth_version=\"1.0\"";

            HttpClient httpClient = new HttpClient();

            httpClient.DefaultRequestHeaders.Authorization = new HttpCredentialsHeaderValue("OAuth", authorizationHeaderParams);
            var httpResponseMessage = await httpClient.GetAsync(new Uri(twitterUrl));

            response = await httpResponseMessage.Content.ReadAsStringAsync();

            return response;
        }

As you can see, we need to create our signature based on the url parameters we are requesting. Twitter demands them to be sorted alphabetically. If you would not sort them this way, the Signature is wrong and your request gets denied by Twitter. We are overloading this Task with the above mentioned cursor to get the next page of our user’s friends list. If you need the GetNonce() and GetSignature() methods, take a look at this post, I am using them for all requests to Twitter. Now that we have this Task ready to make our requests, lets load the complete list. First, declare these objects in your page/model:

        public List<TwitterMentionSuggestionBase> twitterMentionSuggestionsList;
        public long nextfriendlistCursor = -1;
        public string friendsResponseString = string.Empty;
        public string friendsJsonFileName = "twitterFriends.json";

Then we need to create a loop to go through all pages Twitter allows us until we are reaching the rate limit of 15 requests per 15 minutes. I using a do while loop, as this is the most convenient way for this request:

            //to avoid double entries, make sure the list is we are using is empty
            if (twitterMentionSuggestionsList.Count != 0)
            {
                twitterMentionSuggestionsList.Clear();
            }
            //starting the do while loop
            do
            {
                //fetching the Json response
                //to get the first page of the list from Twitter, the cursor needs to be -1
                friendsResponseString = await TwitterFriendsListString(nextfriendlistCursor);

                //using Json.net to deserialize the string
                var friendslist = JsonConvert.DeserializeObject<TwitterFriendsClass.TwitterFriends>(friendsResponseString);
                //setting the cursor for the next request
                nextfriendlistCursor = friendslist.next_cursor;

                //adding 200 users to our list
                foreach (var user in friendslist.users)
                {
                    //using the screen name ToLower() to make the further usage easier
                    TwitterMentionSuggestionsList.Add(new TwitterMentionSuggestionBase() { name = "@" + user.screen_name.ToLower() });
                }

                //once the last page is reached, the cursor will be 0
                if (nextfriendlistCursor == 0)
                {
                    //break will continue the code after the do while loop
                    break;
                }
            }
            //the while condition needs to return true if you would set up a Boolean for it!
            while (nextfriendlistCursor != 0);

As long as the cursor is not 0, the loop will load the json string with 200 entries. The class you’ll need for deserializing can be easily generated. Just set a breakpoint after the call of our Task, copy the whole response string and go to http://json2csharp.com to get your class. The TwitterMentionSuggestionBase class only contains a string property for the name and can be extended for you needs:

    public class TwitterMentionSuggestionBase
    {
        public string name { get; set; }
    }

Of course we need to save the list for further use. The code I use is pretty much the same you’ll find when you search for “saving json string to file” with your favorite search engine and works with Windows Phone 8 and 8.1:

            //saving the file to Isolated Storage
            var JsonListOfFriends = JsonConvert.SerializeObject(twitterMentionSuggestionsList);
            try
            {
                //getting the localFolder our app has access to
                StorageFolder localFolder = ApplicationData.Current.LocalFolder;
                //create/overwrite a file
                StorageFile friendsJsonFile = await localFolder.CreateFileAsync(friendsJsonFileName, CreationCollisionOption.ReplaceExisting);
                //write the json string to the file
                using (IRandomAccessStream writeFileStream = await FriendsJsonFile.OpenAsync(FileAccessMode.ReadWrite))
                {
                    using (DataWriter streamWriter = new DataWriter(writeFileStream))
                    {
                        streamWriter.WriteString(JsonListOfFriends);
                        await streamWriter.StoreAsync();
                    }
                }
            }
            catch
            {

            }

This way, we can easily generate a friends list for our users, based on their Twitter friends. Some points that might be helpful:

  • users with more than 3000 friends will receive an error after reaching this point on our task. Catch this error, but continue to save the already loaded list. Save the latest cursor for the next page and prompt the user to continue loading after 15 minutes to continue loading.
  • if you want to have a list with more details about a user’s friends, extend the TwitterMentionSuggestionBase class with the matching properties.
  • this code can also be used to determine the followers of a user, you just need to change the url parameters as well as the signature parameters according to the matching Twitter endpoint

As always, I hope this is helpful for some of you. Until the next post, happy coding!

Posted by msicc in Dev Stories, windev, 3 comments

How to detect all urls in a string to match Twitter’s requirements (Windows 8(.1) and Windows Phone 8(.1))

url

As you might guess, I am still working on that app I mentioned in my last blog post. As I was diving deeper into the functions I want, I recognized that Twitter does a very well url handling server side.

Like the official documentation says, every url will be shortened with a link that has 22 characters (23 for https urls).

I was trying to write a RegEx expression to detect all the links that can be:

  • http
  • https
  • just plain domain names like “msicc.net”

This is not as easy as it sounds, and so I was a bit struggling. I then talked with @_MadMatt (follow him!) who has a lot of experience with twitter. My first attempt was a bit confusing as I did first only select http and https, then the plain domain names.

I found the names by their domain ending, but had some problems to get their length (which is essential). After the very helpful talk with Matthieu, I finally found a very good working RegEx expression here on GitHub.

I tested it with tons of links, and I got the desired results and it is now also very easy for me to get their length.

Recovering the amount of time I needed for this, I decided to share my solution with you. Here is the method I wrote:

        public int CalculateTweetCountWithLinks(int currentCount, string text)
        {
            int resultCount = 0;

            if (text != string.Empty)
            {
                //detailed explanation: https://gist.github.com/gruber/8891611
                string pattern = @"(?i)\b((?:https?:(?:/{1,3}|[a-z0-9%])|[a-z0-9.\-]+[.](?:com|net|org|edu|gov|mil|aero|asia|biz|cat|coop|info|int|jobs|mobi|museum|name|post|pro|tel|travel|xxx|ac|ad|ae|af|ag|ai|al|am|an|ao|aq|ar|as|at|au|aw|ax|az|ba|bb|bd|be|bf|bg|bh|bi|bj|bm|bn|bo|br|bs|bt|bv|bw|by|bz|ca|cc|cd|cf|cg|ch|ci|ck|cl|cm|cn|co|cr|cs|cu|cv|cx|cy|cz|dd|de|dj|dk|dm|do|dz|ec|ee|eg|eh|er|es|et|eu|fi|fj|fk|fm|fo|fr|ga|gb|gd|ge|gf|gg|gh|gi|gl|gm|gn|gp|gq|gr|gs|gt|gu|gw|gy|hk|hm|hn|hr|ht|hu|id|ie|il|im|in|io|iq|ir|is|it|je|jm|jo|jp|ke|kg|kh|ki|km|kn|kp|kr|kw|ky|kz|la|lb|lc|li|lk|lr|ls|lt|lu|lv|ly|ma|mc|md|me|mg|mh|mk|ml|mm|mn|mo|mp|mq|mr|ms|mt|mu|mv|mw|mx|my|mz|na|nc|ne|nf|ng|ni|nl|no|np|nr|nu|nz|om|pa|pe|pf|pg|ph|pk|pl|pm|pn|pr|ps|pt|pw|py|qa|re|ro|rs|ru|rw|sa|sb|sc|sd|se|sg|sh|si|sj|Ja|sk|sl|sm|sn|so|sr|ss|st|su|sv|sx|sy|sz|tc|td|tf|tg|th|tj|tk|tl|tm|tn|to|tp|tr|tt|tv|tw|tz|ua|ug|uk|us|uy|uz|va|vc|ve|vg|vi|vn|vu|wf|ws|ye|yt|yu|za|zm|zw)/)(?:[^\s()<>{}\[\]]+|\([^\s()]*?\([^\s()]+\)[^\s()]*?\)|\([^\s]+?\))+(?:\([^\s()]*?\([^\s()]+\)[^\s()]*?\)|\([^\s]+?\)|[^\s`!()\[\]{};:'.,<>?«»“”‘’])|(?:(?<!@)[a-z0-9]+(?:[.\-][a-z0-9]+)*[.](?:com|net|org|edu|gov|mil|aero|asia|biz|cat|coop|info|int|jobs|mobi|museum|name|post|pro|tel|travel|xxx|ac|ad|ae|af|ag|ai|al|am|an|ao|aq|ar|as|at|au|aw|ax|az|ba|bb|bd|be|bf|bg|bh|bi|bj|bm|bn|bo|br|bs|bt|bv|bw|by|bz|ca|cc|cd|cf|cg|ch|ci|ck|cl|cm|cn|co|cr|cs|cu|cv|cx|cy|cz|dd|de|dj|dk|dm|do|dz|ec|ee|eg|eh|er|es|et|eu|fi|fj|fk|fm|fo|fr|ga|gb|gd|ge|gf|gg|gh|gi|gl|gm|gn|gp|gq|gr|gs|gt|gu|gw|gy|hk|hm|hn|hr|ht|hu|id|ie|il|im|in|io|iq|ir|is|it|je|jm|jo|jp|ke|kg|kh|ki|km|kn|kp|kr|kw|ky|kz|la|lb|lc|li|lk|lr|ls|lt|lu|lv|ly|ma|mc|md|me|mg|mh|mk|ml|mm|mn|mo|mp|mq|mr|ms|mt|mu|mv|mw|mx|my|mz|na|nc|ne|nf|ng|ni|nl|no|np|nr|nu|nz|om|pa|pe|pf|pg|ph|pk|pl|pm|pn|pr|ps|pt|pw|py|qa|re|ro|rs|ru|rw|sa|sb|sc|sd|se|sg|sh|si|sj|Ja|sk|sl|sm|sn|so|sr|ss|st|su|sv|sx|sy|sz|tc|td|tf|tg|th|tj|tk|tl|tm|tn|to|tp|tr|tt|tv|tw|tz|ua|ug|uk|us|uy|uz|va|vc|ve|vg|vi|vn|vu|wf|ws|ye|yt|yu|za|zm|zw)\b/?(?!@)))";

                //generating a MatchCollection
                MatchCollection linksInText = Regex.Matches(text, pattern, RegexOptions.Multiline);

                //going forward only when links where found
                if (linksInText.Count != 0)
                {
                    //important to set them to 0 to get the correct count
                    int linkValueLength = 0;
                    int httpOrNoneCount = 0;
                    int httpsCount = 0;

                    foreach (Match m in linksInText)
                    {
                        //https urls need 23 characters, http and others 22
                        if (m.Value.Contains("https://"))
                        {
                            httpsCount = httpsCount + 1;
                        }
                        else
                        {
                            httpOrNoneCount = httpOrNoneCount + 1;
                        }

                        linkValueLength = linkValueLength + m.Value.Length;
                    }

                    //generating summaries of character counts
                    int httpOrNoneReplacedValueLength = httpOrNoneCount * 22;
                    int httpsReplacedValueLength = httpsCount * 23;

                    //calculating final count
                    resultCount = (currentCount - linkValueLength) + (httpOrNoneReplacedValueLength + httpsReplacedValueLength);                    
                }
                else
                {
                    resultCount = currentCount;
                }
            }
            return resultCount;
        }

First, we are detecting links in the string using the above mentioned RegEx expression and collect them in a MatchCollection.

As https urls have a 23 character length on t.co (Twitter’s url shortener), I am generating two new counts – one for https, one for all other urls.

The last step is to substract the the length of all Match values and add the newly calculated replaced link values lengths.

Add this little method to your TextChanged event, and you will be able to detect the character count on the fly.

As always, I hope this is helpful for some of you.

Happy coding, everyone!

 

Posted by msicc in Dev Stories, windev, 0 comments

Getting productive with WAMS: How to call Twitter REST API 1.1 from a scheduled script

WAMS.png

Like I promised in my first post about Windows Azure Mobile Services, I will show you how to call the Twitter Rest API 1.1 from a scheduled script. The documentation of the http request object does only use Twitter API 1.0 (which is no longer available).

First, you will need a Consumer key and a Consumer secret for your app. Just go to dev.twitter.com, register with your Twitter account and then add a new application.

The second thing you will need, is the so called Access token and Access token secret. Both are user dependent, without them Twitter will give you an error that your app is not authorized to use this account for anything on Twitter.

There are several ways to obtain these values. As I am registering the user within my phone app, I am uploading these values from phone and store it in my Mobile Services database.

To generate the requested data, we need several additional data for our request to Twitter:

  • a timestamp for the oAuth Header and the signature string
  • a random number to secure the request (= nonce)
  • an oAuth signature (signed array of the user’s data)
  • a HMAC encoded Hash string

These data is used for our request to Twitter.

Let’s start with the “simple” things:

generate Timestamp:

//generating the timestamp for the OAuth Header and signature string
var timestamp  = new Date() / 1000;
timestamp = Math.round(timestamp);

generate nonce

function generateNonce() {
    var code = "";
    for (var i = 0; i < 20; i++) {
        code += Math.floor(Math.random() * 9).toString();
    }
    return code;
}

oAuth signature

//generating the oAuth signatured array for the Twitter request
function generateOAuthSignature(method, url, data) {
    //remove query string parameters
    var index = url.indexOf('?');
    if (index > 0)
        url = url.substring(0, url.indexOf('?'));

    var signingToken = encodeURIComponent(ConsumerSecret) + "&" + encodeURIComponent(twitterAccessTokenSecret);

    var keys = [];
    for (var d in data) {
        if (d != 'oauth_signature') {
            //console.log('data:', d);
            keys.push(d);
        }
    }

    keys.sort();
    var output = "GET&" + encodeURIComponent(url) + "&";
    var params = "";
    keys.forEach(function (k) {
        params += "&" + encodeURIComponent(k) + "=" + encodeURIComponent(data[k]);
    });
    params = encodeURIComponent(params.substring(1));

    return hashString(signingToken, output + params, "base64");
}

generate the HMAC encoded hash string

//generate Hash-string, encoded in HMAC-SHA1 as required by Twitter's API v1.1
function hashString(key, str, encoding) {
    //console.log('basestring:', str);
    var hmac = crypto.createHmac("sha1", key);
    hmac.update(str);
    return hmac.digest(encoding);
}

Now we have prepared all of these functions, we are well prepared to call the Twitter API. In this example we are calling the user’s profile data:

function requestToTwitter()
{

    //the url declaration has to be in this function to make the request working!
    //declaring it in another function would cause an error 401 from Twitter's API
    url = 'https://api.twitter.com/1.1/users/show.json?user_id=' + twitterId;

    //generate data for sending the request to Twitter
    //this is the data used in the signature string as well as in the Authorization header
    var oAuthData = {oauth_consumer_key: ConsumerKey, oauth_nonce: nonce, oauth_signature: null, oauth_signature_method: "HMAC-SHA1", oauth_timestamp: timestamp, oauth_token: twitterAccessToken, oauth_version: "1.0"};
    var sigData = {};
    for (var k in oAuthData) {
        sigData[k] = oAuthData[k];
    }
    sigData['user_id'] = twitterId;

    var sig = generateOAuthSignature('GET', url, sigData);
    oAuthData.oauth_signature = sig;

    var oAuthHeader = "";
    for (k in oAuthData) {
        oAuthHeader += "," + encodeURIComponent(k) + "=\"" + encodeURIComponent(oAuthData[k]) + "\"";
    }
    oAuthHeader = oAuthHeader.substring(1);
    //very important to not miss the space after OAuth!
    authHeader = 'OAuth '+oAuthHeader;

    var reqOptions = {
            uri: url,
            headers: { 'Accept': 'application/json', 'Authorization': authHeader }
    };

    var httpRequest = require('request');
        httpRequest(reqOptions,callback );

}

var callback = function(err, response, body) {
    //console.log("in requestToTwitter = callback"); 
            if (err) {
            console.log(err)
            } else if (response.statusCode !== 200) {
                console.log("from twitter callback " + response.statusCode + " response: " + response.body);
            } else {
                var userProfile = JSON.parse(body);
                UserIdFromTwitter = userProfile.id;
                twitterScreenName = userProfile.screen_name;

}
}

You may have noticed that there are several variables that are not declared within these functions. Just declare them globally in your scheduled script.

You can read more about the oAuth authorization process at http://oauth.net/.

There are more services out there that use the oAuth process, so you should be able to convert this for other requests, like getPocket.com (formerly Read It later) and others.

As always, I hope this post was helpful for some of you.

Happy coding!

Posted by msicc in Azure, Dev Stories, 0 comments