New York City dataset enhancement tutorial for HP Haven OnDemand

Background

The HP Haven Big Data platform harnesses 100% of your data—structured and unstructured—to inform every decision and help you capitalize on opportunities and solve problems. Available on-premise or in the cloud, Haven offers Big Data analytics and next gen applications at unmatched speed and scale.

Through a mix of fun and real world challenges, HP is inviting the TopCoder community to learn how to build the next generation of Big Data and analytics apps using the Haven OnDemand platform. We hope that this series will be interesting, challenging, and rewarding for developers of all levels that are looking to gain valuable new skills and experience.  You can find the latest Topcoder challenges related to the HP Haven Big Data Platform here:

http://hphaven.topcoder.com/

More information about the complete HP Haven offering can be found at the HP Haven web site:

http://www8.hp.com/us/en/software-solutions/big-data-platform-haven/

New York City dataset enhancement tutorial for HP Haven OnDemand

This is part of a series of challenges for Topcoder members to engage with the Vertica database and analytics platform, while at the same time tapping into the power of the broader Haven platform through APIs such as those from HP IDOL OnDemand. 

In a previous challenge, we installed a local version of the Vertica Analytics platform. Now, we’re going to build on that by asking members to create an application for use in support of a developer centric tutorial which analyzes and enhances a data set using IDOL OnDemand’s Query Search and/or Web Connector functionality and then saves the data to Vertica for further analysis.  The Vertica installation instructions from the previous challenge are attached to this competition.

Challenge Instructions/Requirements

Here are the steps to participate in the New York City dataset enhancement tutorial for HP Haven OnDemand challenge:

1. You’ll be creating a command-line application which performs the following functions:
   1. ���Reads data from the Mapped In NYC csv file attached to this challenge.
   2. Uses IDOL OnDemand Query Text API call to perform a Patent Search on each of the companies AND / OR uses the IDOL OnDemand Connector API to load the Job Openings for each of the companies into an IDOL OnDemand Index. OPTIONAL would be to also use the Entity Extraction API to extract things like people, addresses, professions etc. or even your own custom entity type that can also be added into Vertica for a more valueable dataset creation.
   3. Loads the company data from the original report as well as the Patent or Job information for the companies into your local copy of Vertica.
2. You should include data definition scripts which will create the tables that you need in Vertica.
3. You have creative license about what language/platform to use.  
4. Sample Java and Python code to connect to IDOL OnDemand is attached to the challenge but you don't have to use these languages.  Sample code can also be found on the IDOL OnDemand Community site.
5. This is a tutorial challenge. Your code should be clear and well documented.
6. You should produce a blog post about your application. The blog post is an integral part of this challenge, and may even be featured on the HP developer community. 
7. You should produce a screensharing video which explains your code and how to set up and connect to a Vertica database.
8. We're currently running a Sweepstakes challenge which walks through the Vertica setup on a local VMWare instance.  We're also attaching a Vertica lab manual which describes how to add users, create schemas, and load data into the system.  It assumes, however, that you have the Vertica Virtual Server instance installed and locally available.

Getting started up with IDOL OnDemand

Before you can use the API’s you’ll need to sign up for an IDOL OnDemand developer account:

http://www.idolondemand.com/signup.html

Please indicate that you heard about IDOL OnDemand through [topcoder] in the “How did you hear about IDOL OnDemand?” field:

Once your account has been verified you’ll be assigned a developer account and API Key that will allow you to make API calls.   Complete information about available IDOL OnDemand API’s can be found here: 

https://www.idolondemand.com/developer/apis

You’ll need to register for a developer account with HP in order to get access to additional Try functionality in the API console.  Use of the APIs is free and restricted to non-commercial use at this time. Commercial use and pricing will be announced in the near future.

Before you compete in an IDOL-related challenge on [topcoder] please create a topcoder-specific key in your IDOL OnDemand Account.  You can do this by Clicking on Account->API Keys from the developer home page.

Simply generate a new key and rename it to “topcoder” as shown above.   This should be the key that you use in [topcoder] challenge completion.  This will also give you visibility to Preview API’s which may not yet be in public release.  Employees and direct and indirect subcontractors of Hewlett-Packard Company and its subsidiaries and other affiliates (“HP”), and employees and direct and indirect subcontractors of HP’s partners (including TopCoder and its affiliates) are not eligible to participate in the challenge.

You should be all set!

Final Submission Guidelines

Submission Guidelines

1. You should include the source files for your application.

2. Your submission zip should also include sql/ddl scripts to create your database structures.  

3. Please a submission.txt file in the root folder of your submission zip with links to your blog post and video tutorial.

4. Please include instructions on how to build and deploy your app.

5. You can obtain an IDOL OnDemand license key by following the steps outlined in the "Getting Started with IDOL OnDemand" section above.  There is no cost for a developer account.

Employees and direct and indirect subcontractors of Hewlett-Packard Company and its subsidiaries and other affiliates (“HP”), and employees and direct and indirect subcontractors of HP’s partners (including TopCoder and its affiliates) are not eligible to participate in the challenge.