Job Fiction

• BACK

Data Ingestion

We choose Indeed API to acquire jobs posted across United States. Out of all the APIs evaluated (LinkedIn, Glassdoor, Monster and Indeed), Indeed API offered flexible API and convenient rate limits. Indeed API limits to the first 10000 results, leading to our decision to run batches based on the zip codes across US.

A Cron job runs the data acquisition script daily to pull new jobs and store the data as is into the MongoDB store "Job Fiction". Since the results from Indeed API are well formed JSON, it was an easy decision to store the raw data in MongoDB. We started collecting all the jobs without any filtering and later filtered only "data" related jobs. The resultant job posts had job titles varied from truck drivers to data scientists.

Quick numbers and facts on the data acquisition process

• 1.5 Million job posts were accumulated in JSON format - ~ 4 GB volume of data
• Jobs pulled daily and stored in MongoDB and synced up with ElasticSearch instance for data exploration
• No filtering - all jobs with titles varying from truck driver to CEO are pulled
• Less than 20% of all jobs collected contained the word "data"

• BACK

• BACK

Data Ingestion

We choose Indeed API to acquire jobs posted across United States. Out of all the APIs evaluated (LinkedIn, Glassdoor, Monster and Indeed), Indeed API offered flexible API and convenient rate limits. Indeed API limits to the first 10000 results, leading to our decision to run batches based on the zip codes across US.

A Cron job runs the data acquisition script daily to pull new jobs and store the data as is into the MongoDB store "Job Fiction". Since the results from Indeed API are well formed JSON, it was an easy decision to store the raw data in MongoDB. We started collecting all the jobs without any filtering and later filtered only "data" related jobs. The resultant job posts had job titles varied from truck drivers to data scientists.

Quick numbers and facts on the data acquisition process

• 1.5 Million job posts were accumulated in JSON format - ~ 4 GB volume of data
• Jobs pulled daily and stored in MongoDB and synced up with ElasticSearch instance for data exploration
• No filtering - all jobs with titles varying from truck driver to CEO are pulled
• Less than 20% of all jobs collected contained the word "data"

• BACK

• BACK

Solution Architecture

The job data flow from Indeed API is ingested with a CRON job. A simple registration on Indeed API web services gives programmatic access to millions of jobs. The rows are then stored in a single MongoDB NoSQL database. A second job is run to extract keywords representing skills from each job summary. This task is handle by MAUI Indexer, a supervised keyword extraction tool that leverages a controlled vocabulary of skills. The data is then cleaned for some feature extractions that will serve in the job recommendation model.

On the right side of the diagram is where job descriptions submitted by the user are integrated. Keyword extraction program is on those inputs and the calculated keywords are presented back to the users. After users filter and input weights on each skill, those are passed to the recommendation model.

Using the 2 inputs from users - job posts and user preferences - recommender searches for the best job posts that fit contextually to the job descriptions and also matches user’s preferences. In the backend, model is built on the features extracted from the pre-processed text by removing stop-words, stemming text and finding the best n-grams. Our model uses LDA (latent Dirichlet Allocation) for topic modeling and LSI (Latent Semantic Indexing) based similarity indices using the features extracted from the text i.e. TF-IDF, bag-of- words, keywords and best n-grams. Recommender first identifies the clusters where jobs fit in and then find contextually similar jobs within the clusters using topic specific dictionaries and keywords. Finally, recommender arrives at the two scores - job context relevancy score and user preference match score which are combined as model score presented on the results. This score tells how confident model thinks a particular job post is closer to the user’s interests. In term of logistics, model itself is persisted as files on disk and served through Flask microservices The actual job details such as job ids, location, title, keywords are stored on single node ElasticSearch instance.

• BACK

• BACK

Technology Stack

• BACK

• BACK

Keyword Extraction using MAUI

MAUI stands for Multi-purpose automatic topic indexing. It’s a java library and its core is the machine-learning toolkit Weka. Below steps explains how job fiction uses MAUI for controlled keyword extraction to identify skills from job posts.

1. Tokenize candidate keyphrase using a thesaurus as a control dictionary. Following are the controlled dictionaries used
• 2012 ACM Computing Classification System
• O*NET (2016) occupational skills
• word2vec embeddings from the job posts corpus
2. Compute features
• TF-IDF: Term Frequency Inverse Document Frequency
• First Occurence
• Node Degree
• Length
3. Train a model
• Using Naive Bayes as a supervised classifier
• Leveraging a set of labels acquired from JobScan.co, a tool to extract keywords from resume and job ads.
4. Keyword extraction
• Use the generated model to tag keywords. The associated likelihood and the keyword categories are also returned.

• BACK

• BACK

Text Pre-Processing

Filter Noise

Out of the 1.3 Million jobs we acquired from Indeed, there were 250K jobs with the word "data" in either job title or job description. We filtered "data" related jobs based on the assumption that data science related job will definitely contain the word "data". However, based on cursory analysis on the filtered jobs (and multiple iterations of training and validating the model) we found there were too many irrelevant jobs related to office administration, delivery services, care providers etc. which did not fit our expectations.

To filter the noise, we used inclusion and exclusion patterns in the job description and job title. The patterns were based on the research by Data Science Central team on ~10K LinkedIn professionals with titles related to Data Science and categorized 400 job titles for data scientists. We formed inclusion patterns based on the cleaned job title from the research and also formed exclusion patterns analyzing job titles that clearly did not fell into data science related job titles. After filtering the noise from master job posts database, we had 85K job posts related to data science field i.e. ~6.5% of job posts collected.

Text Normalization

After analyzing sample job posts, we observed that companies post job descriptions in different formats and varying lengths and different lengths. Most of the job descriptions have - company information (40%), job responsibilities (58%), qualification (50%) and skills (79%). Length of job posts followed typical zipf distribution. We concatenated job title with job description for the entire analysis.

We applied following text normalization techniques before building features:

• Removed non-alphanumeric characters including punctuations (except +, ., -)
• Text converted to lower case
• Removed URLs and email ids
• Removed stop words (used smart list stop words, NLTK stop words and our own stop words based on the analysis)
• Stemming (Snowball stemmer) to reduce words to their root form

We built text pre-processing module with the ability to enable/disable any specific normalization technique. Based on multiple iterations and analysis, our current model does not apply stemming as some of the keywords were lost during the process. After applying the normalization techniques, size of the relevant job posts data came down to 250M from 400M.

• BACK

• BACK

Modeling & Indexing

Feature Extraction

The primary goal of the model is to predict job title/job class given set of job descriptions and respective features. Since the job titles for data science related jobs vary from analyst to C level and job posts under similar titles have varying responsibilities and qualifications, our attempt is to classify the jobs based on the job descriptions and skills required instead of just job titles. Our expected job classes are: Data Analyst, Data Engineer, Machine Learning Engineer, Research, Data Visualization, Research, Chief level positions, Data Scientists, Data Architect etc. Given a set of job posts, our model should be able to classify job post as x% of class-1, y% of class-2 etc. This is similar to classifying the reviews of a restaurant based on food, ambience, service,

Due to unavailability of training data set we resorted to unsupervised learning techniques such as Latent Dirichlet Allocation (LDA) to identify the job classes. We looked for features to help LDA to come up with topics closer to our expectations. We went through multiple iterations to analyze the features or inputs required for predicting decent job class.

The key features we looked at

• 1. Skill Identifiers (Keywords): We used controlled keyword extraction using MAUI (Multi-purpose Topic Indexing) algorithm to extract key words and phrases based on the O*NET and ACM taxonomy classification. We added additional skills to the dictionary based on the big data and data science related technologies. The skills extracted were one of the important features to the model to classify job posts. We explored alternative approaches before banking on keywords from MAUI such as Named Entity Extraction (NER) to identify skills, Word2vec embedding to look at local context of skills (such as skills similar to Python). We could not get right keywords from NER due to no training data set and general corpus trained data sets did not work. Word2vec embedding are promising to identify similar skills but does not identify a skill. We used Word2Vec embedding to form a vocabulary for MAUI controlled keyword extraction.
• 2. Bag-of-Words: Each sentence and the entire job post was transformed to a bag-of-words model to represent each job post as a set of words and frequency irrespective of grammar and word order.
• 3. TF-IDF: Each job description is converted into a vector space model base don term frequency and inverse document frequency scores. Minimum thresholds are set to remove any common words even after removal of stop-words.
• 4. Best n-Grams: From the job posts corpus, we found best n-grams with n ranging from 1 to 5. This feature was part of improvements we identified after initial iterations of model building exercise. Best n-grams has immensely helped LDA topic clusters to make more sense.

• BACK

• BACK

Recommendations and Results

Recommender

Job Fiction recommender uses pre-built indices during training to compare and retreive the best results from the job posts database based on contextual similarity of job descriptions entered by the job seeker and his/her preferences. Following steps summarize the action behind recommendations.

• Gather and pre-process user entered job titles and job descriptions along with user preferences tagged - must have, nice-to-have and exclusions.
• Find topics (or clusters) where user entered [Job Title + Job description] fits into the trained LDA topic clusters. This is based on the jobs pre-trained on the LDA model. This is equivalent to semantic searching (contextual search)
• Based on the scores of top 3 topics, recommender retrieves the job posts in those topics. Score of each job post retrieved = (topic_score) * similarity_score(compared with [Job Title + Job description])
• From this list of jobs, check for each job how many of them have user preferred skills. Apply term weights to determine score.
• The list of top N job ids are returned to the front-end interface through Python Flask micro-service
• The client interface queries ElasticSearch to fetch job details - title, location, job class, URL and necessary charts to show how job post returned matches the skills.

Results

Results page has the ability to scan through jobs and compare jobs in the same view. This was done by using collapsible rows allow the user to show or hide the details of the job. The heading row was set to include the job title, company and location. After, the job classification and confidence scores were added.

The collapsible section includes a link to the job online, a table showing the counts for “Must Have” and “Nice to Have” and a graph to visually show the job skills. Sort to add to the results and a search field to allow additional filtering on search results. The search only filters on information in the heading row of the job in the search table - not the actual job content itself.

Graphs are an important feature to Job Fiction because no other job search site (to current knowledge) has a way to visualize a job. While bar graphs are some of the simpler forms of visualizations, it is an appropriate choice for the data at hand. The job terms or “tokens” parsed from the job description are categorized and the counts for each category fed into the graph. The counts of tokens in each category make the primary graph for the jobs.

For added dimension, clicking on the graph bars reveals a second graph with counts of secondary token levels. This gives further information and ideas to the job applicant of what job features are really important to them in an organized way.

• BACK

• BACK

Evaluation of Model

There really is not a standard way to evaluate retrieval effectiveness of a job search engine, or a search engine of any type for that matter. The results can only be subjectively compared to what an individual is expecting, or what is done by similar systems.

Evaluation

There are two particular problems in evaluating Job Fiction’s model.

• Unlike commercial job search engines (LinkedIn, Indeed, Dice, etc.), Job Fiction's database is not a live database and the job posts returned may already be expired.
• The retrieval logic of the commercial job search engines is not known, and each has its own advance search options available to users. To say the jobs returned from two different job search engines are not the same is only acknowledging that each search algorithm is different.

One search engine that is similar to Job Fiction is JobScan.co. Though the main goal of JobScan.co is to evaluate the match between a resume and a job description, it offers a list of jobs similar to the job description submitted. Submitting the same job description to Job Fiction and to JobScan.co result in different jobs; this is expected since Job Fiction queries it’s own dataset. Subjectively comparing the job titles reviewed, it is possible to suggest Job Fiction returns better matches, but since the goal of Job Fiction is to ignore job titles as an indicator of an appropriate job match this too is not an appropriate evaluation.

Feedback

From the minimal usability testing conducted,

• There is general excitement for the results returned. The job results are interesting to the user, and some have titles the user would typically dismiss and never consider.
• Users were at first confused with the usability and functionality until realized just how simple it was. It was later complimented for being so simple and to-the-point, saying they were not used to such a straight-forward design.
• The job classification in the results is a little less useful, especially if the three job descriptions entered are already very similar.
• Even though the jobs returned matched the user’s interest, many still included the job tokens the user explicitly placed in the Exclude list, suggesting the weights used in the model’s “Must Have,” Nice to Have” and “Exclude” lists were not effective.

• BACK

• BACK

Learnings and Future Work

Learnings

1. Project Management
• Outline the design at the outset of project to avoid issues with integration of components.
• When the team is dispersed, plan to more often than once a week.
• Start working on the user interface and home page from the beginning.
2. ML Modeling
• Capture baseline of the model as early as possible
• Plan to have minimal training data set even though it's expensive.
• Assess quality of the data and filter irrelevant data to build a better model and results
• Engage users early into the game to make quantitative evaluation of the model

Future Work

• Automatic Topic Labeling: Instead of manually assigning topic labels, utilize word2vec embeddings to automatically assign topic labels.
• Live database: Job Fiction model is based on 1.3 Million jobs acquired and current job results are not live. Making the database real time will give live results.
• Better weights on tokens: Adjust weights on tokens such that “Must Have,” “Nice to Have” and “Exclude” all have noticeable influence on the results.
• Advanced Search Options: Add filters for location, company types, and other general job features.
• Third Party Data Integration: Working with a company like GlassDoor would allow us to query real-time datasets of jobs and give us access to additional data like salary, company size, job titles that we cannot access with the public APIs.

• BACK