A Lithium brand
A more neutral topic curation algorithm
SarahE
7
Kudos
149
Comments
May 16, 2016

 

Monday, Gizmodo published an article about the curation practices behind Facebook’s Trending module, and to what extent a curator’s personal biases affect what’s shown to Facebook’s billion-plus active users. (As you can imagine, this has caused some controversy.) Although Gizmodo focused on the role of human curators, those of us who work with algorithms and machine learning have had to confront the fact that biases can end up deeply encoded into supposedly objective systems, including Klout’s own topic classifier -- the system that identifies your Expert topics, as well as the relevant articles for the Explore tab. Here’s a quick roundup of how we in Klout’s Data Science team think about keeping the topic system as unbiased as possible.

 

Klout’s topic classifier in 30 seconds or less

The major inputs to our topic system are:

  1. The classifications we’re applying:
    1. An ontology of nearly 10,000 human-curated topics
    2. An underlying dictionary of over a million named entities and concepts
  2. The data being classified, which includes:
    1. Social media profiles (to be analyzed for topic expertise and interest)
    2. URLs published on social media and elsewhere (to be analyzed for topical content and served in the Explore tab)

 

Even without getting into the weeds of the pipeline that brings those two types of input together -- more on that below -- critical readers will already be able to spot a few areas where we’re vulnerable to bias. Let’s walk through them.

 

Any classification system contains value choices

Should “Autism” be placed under “Diseases” or “Neurology”? Is “the Tea Party movement” distinct from “conservative politics”? Is “Wizard Rock” really a thing? How we answer these questions shapes the experience of our users, and inevitably gives our ontology a point of view.

 

Longtime users may remember the early days of Klout, when topics were a, shall we say, messy combination of user-submitted tags and data-mined concepts. As the Data Science team worked on regularizing and improving the ontology, we’ve relied on the following principles:

 

  1. The ontology is a living thing; we should always have tools in place for updates
  2. The ontology should always have up-to-date guidelines defining:
    1. Scope -- what portion of the world we’re describing. (In our case, as much of it as possible.)
    2. Granularity -- at what level of detail we’re describing it. (Do we need to include every actor? Television show? Heavy metal subgenre?)
    3. Voice -- the tone we use in describing it. (Do we use scientific names? Full legal names for persons? Slang?)
  3. A little redundancy won’t hurt  -- our system can support topics with some conceptual overlap, so err on the side of inclusion. (For example, both “Gun Rights” and “Gun Control” are topics in the Klout ontology.)
  4. Users are the best source of feedback -- our users have a broader range of perspectives than we do; make it easy for them to alert us to problems

 

Even so, any time your application or audience changes, it’s important to reassess your classification scheme. One major flaw in Klout’s topic ontology is that it was developed for a U.S. audience, and still needs significant work for other countries and languages.

 

Staying alert for sins of omission

In addition to the human-curated topics in the ontology, we also use a dictionary of concepts and entities derived from Freebase. Freebase is a widely-used resource in the data science world, but “widely-used” is not the same thing as “perfect”, by any means. The biggest issue with Freebase is what it leaves out; like Wikipedia, it was collectively sourced, and like Wikipedia, it’s biased toward the interests of its editors, and sparse in some areas like cosmetic products and fashion terms, requiring us to develop ways to supplement the dictionary. The moral of the story: it pays to look critically at any pre-packaged data set you plan to use.

 

Boosting inclusivity

Next, let’s consider the URLs we collect for the Explore tab. The majority are URLs that have been shared on social media, which means they are dominated by the topics most discussed on social media: politics, celebrity news, music, etc. What we sometimes call “niche topics”, like molecular biology, or Wicca, or wheelchairs, naturally are present in fewer URLs. Does that count as a bias? It’s unclear, but it’s not a good end-user experience and risks making some users feel marginalized. As a result, we’ve had to develop backup strategies to increase coverage for less common topics.

 

The fuzzy line between human bias and business logic

One of the more ironic tidbits in Gizmodo’s article was that Facebook’s curators were told to suppress news about Facebook -- that is, to interfere with the Trending algorithm to avoid the appearance that Facebook was interfering with the Trending algorithm. But that kind of decision is probably familiar to the product managers in the audience, whose goal it is to preserve the user experience. Similarly, a discovery feed like our Explore tab might recommend porn, or spam, or hate speech, and need to be tuned or overridden. To make it even more complicated, the definition of porn, or spam, or hate speech may change from region to region. Keeping those decisions from being made inconsistently or thoughtlessly is really difficult, but our approach has been to define a single owner who both documents the rules and is accessible to discuss individual cases. As others have pointed out, Facebook’s mistake may not have been having curatorial tools, but isolating the employees using them.

 

Fine, but what about the actual topic algorithm?

Eagle-eyed readers will have noticed that we haven’t touched on the nuts and bolts of how Klout’s topic system actually assigns topics. The challenges of data modeling and debugging machine learning algorithms are pretty well surveyed elsewhere, and how we handle those challenges at Klout would require a dedicated blog post. However, there’s less discussion of how to handle human biases when collecting training or validation data -- how people’s points of view get encoded into the data a given algorithm is trying to approximate. The two approaches often recommended could be described as micromanaging versus crowdsourcing; either a) have an in-house process that includes well-defined guidelines, trained judges, and a reconciliation process for disagreements, or b) have lightweight guidelines but a large number of judges, in the hopes that individual biases will be muted. There are tradeoffs to either approach; our team has recently been relying mostly on in-house validation data, mostly because it’s friendly to our development schedule. But what’s more important, in our experience, is that the potential weaknesses of the training/validation data are known and discussed and documented ahead of time, so that they can be distinguished from problems with the model itself.


No system is perfect, and keeping out bias takes continual work. Although a focus on documentation, consistency, and validation will take you a long way, the very best defense against unintentional bias is a diverse team, who can bring multiple points of view. Want to come work with us?

 

 


IMG_750.jpgSarah Ellinger is the Lead Data Analyst for Klout/Lithium’s Data Science team. She is responsible for overseeing the content of the topic ontology, as well as monitoring the performance of the topic classification system. Sarah attended U.C. Berkeley’s School of Information and has over a decade of experience in taxonomy and web content classification at tech companies large and small. She can be found on Twitter discussing information science and Game of Thrones spoilers @sarahellinger.

149 Comments
admin88
Not applicable

Your article makes me more experienced and impressed, I hope you will have more good posts in the near future to share with readers. geometry dash

jabbar_hussain1
Not applicable
see more news about education http://studyabroad.resultonline.pk/search.php
markewalker
Not applicable

Do you want to proofread my paper but you do not know exactly how to do this properly. You should not worry as you can hire expert writing services that are working online. 

gusulazi
Not applicable


They just wanted to stand out because they didn't want to fit in what Avery considered to be "robots" and conform, and they were well versed in history and how scary it is to disassociate and forget who you are, is as it happened to them before Hunks gay movies. To avoid that, as they entered middle school, Avery began to listen to metal grunge and punk music and began wearing ripped jeans, crop tops and they colored and dyed their hair frequent colors and sometimes Avery even shaved part of her hair off to the point of only having it barely being there for mostly visual and sensory reasons in addition to wanting to stand out. Avery was a full-blown punk and goth. Also, Avery loved punk culture and loved it after seeing one of their friends wear that style, that encouraged Avery to "liberate" themselves and their body, and Avery's parents were OK with them wearing those types of clothing as Avery's parents just like Avery were non-binary and Avery became non-binary at age 12 and before that Avery still considered themselves to be female and preferred she/her pronouns until 12 Interracial gays

Avery was extremely ticklish and their parents knew that, and Avery's parents did everything to raise Avery the way they always dreamed of raising their first child. Avery's parents didn't dislike their behavior, but whenever They would misbehave or do something to seek attention, Avery would get tickled.

 

 

 

 

 

 

Mobilepricy
Not applicable

This is really an awsome blog because i found all my required information here. I recently found another blog with same content which is updated on daily basis. You can check it here. Apple Iphone 12 Price and Specification 

raghuveerc1
Not applicable

Awesome information thanku so much for Sharing Movie Download 

raghuveerc1
Not applicable

Its a awesome blog and its really very very very helpful blog 

 

My all request complete this forum and its all information great MBA Full Form 

petersmit
Not applicable

Fixing of Error Code 30 of HP Printer

HP printer is a globally used printer. Hp is a reputed brand that has many varieties of products. Regardless of being reputed with multi - functionality features, HP Printers comes across with many technical errors.  Hp Printer non - activated error code 30 is one of them. This is a printer non activation error that can be solved in very simple steps. Here we will tell you how to resolve this error code 30 in few simple steps. In case if you are still facing some issues, then plc contact HP Printer helpline number+1-833-284-2444. Experts will help to solve your concerns in a few minutes.

To know more click here

phamyen
Not applicable

Thank you for sharing this article. It is really useful, hope you can share more other articles.

five nights at freddy's unblocked

priny15
Not applicable
I am thankful for the article post.Looking forward to visit more. voir film complet
lgarvindgupta
Not applicable

Proprio per questo motivo, ho deciso di condividere con voi divese frasi di Buon Compleanno da dedicare alle persone più care, sulla bacheca di facebook, su un bigliettino o a voce,
per rendere la loro giornata più allegra! Preparati a far sorridere i tuoi amici con queste fantastiche battute.
Queste sono solo alcune delle frasi da dedicare, ma ti ricordo che nel menù in alto ne puoi trovare tante altre in base al tipo di persona a cui devi fare gli auguri..

admin88
Not applicable

I don't know what to say really what you share is so good and helpful to the community, I feel that it makes our community much more developed, thanks.  online jigsaw puzzles    

jessicaah
Not applicable

Australian Assignment Help provides best quality assistance to students at reasonable prices. Get my assignment help services in Australia from the quality writers of Australian Assignment Help and get assignment writing help in Oman from the experts of Student Life Saviour. 

rohnnylaw
Not applicable
We offer the best Law essay services in the UK with exceptional original written work from British native writers. UK professional law experts who have also been providing Law coursework help for years. Order with us and get good grades in your university exams!
 
nobita88
Not applicable

I don't know what to say really what you share is so good and helpful to the community, I feel that it makes our community much more developed, thank you. online jigsaw puzzles

Raghubro123
Not applicable
Thanks for sharing this helpful & wonderful post. I really appreciate your hard work. This is very useful & informative for me. Thanks for sharing with us. Thanks a lot. Regards
 Adviceduniya and helpgurugroup
Raghubro123
Not applicable
Techlinics
Not applicable

Very nice join free computer tips, hacking, html, java, css, and tricks in Techlinics Visit Site

And

Free digital marketing course

For Computer Gyan

 

Techlinics
Not applicable

Chole bhatore best recipe

Lucknawi Zayka

lindarose11
Not applicable

This is extremely helpful info!! Very good work. Everything is very interesting to learn and easy to understood. Thank you for giving information.

bonk io

emmarobet
Not applicable

I am truly thankful to the holder of this web page who has shared this enormous piece of writing at at this place. president leather jacket

SophiaAlex
Not applicable

Excellent and nice post. It will beneficial for everyone. Thanks for sharing such a wonderful post.It is extremely helpful for me.

Batman Arkham Knight Nightwing Leather Jacket 

inhouse
Not applicable

Breast cancer Philippines is still the most common cancer affecting women. In 2018, there were 24,798 new cases of breast cancer among women in the Philippines. That is 31.4% of all new cancer cases in women for that year, making it the top cause of death among Filipina women. There are many options for treatment of breast cancer in the Philippines. Many local surgeons are trained and experienced when it comes to treating the condition. However, a common concern that breast cancer patients are the look of their breast after treatment, especially in surgeries like a mastectomy.

willwon155
Not applicable

 , fax machine servicing & repairs and photocopier service & repairs. Our aim is to provide you with Fast onsite Service and Repairs as well as providing you with 100% backup on our service and parts. 

robertjake8636
Not applicable

Thank you for organizing such wonderful info. Students can now make their life easy and hassle-free by opting Assignment Help service of GoAssignmentHelp portal. It is one stop gateway for success in the academic field. You can ask for administrative law assignment help online. We will make it for you at affordable price.

jamessmith
Not applicable

www.trendmicro.com/getmax Maximum Security uses advanced artificial intelligence technology to provide robust, multi-device protection.It is compatible with various devices like PC, Mac, Smartphones, windows, and laptop. With its ultimate features trend micro download and install process is secure that protects your devices and internet from ransom, malware, spyware, virus, and other threats. Examine buying Trend micro from Trend Micro if you are going to buy a new device for your home, official or commercial use. Trend micro geek squad is a 100% security provider antivirus that removes all types of threats from your devices as well as from the network.

raghuveerc1
Not applicable

Awesome information thanks for share this information sir helpgurugroup.com most usefull information avalable in english wealthpage.org article

raghuveerc1
Not applicable

Awesome information thanks for share this information sir helpgurugroup.com

raghuveerc1
Not applicable

most usefull information avalable in english wealthpage.org article

lesacote
Not applicable

Thank you for the wonderful blog on curation algorithm. The supply chain management assignment help appreciates your effort. 

lesacote
Not applicable


Thank you for the wonderful post on curation of algorithm and the supply chain management assignment help  admires this post! 

adjonline
Not applicable

Finding the platform that offers judgments, educational material and other contents related to the law under one umbrella is not that much easy. But ADJ Online is a new and effective way to read the judgments, records and other related tasks anywhere and anytime you want. With the help of LawSoftware, aspirants may readily assist the latest as well as the oldest judgments of Lucknow High Court.

 

To know More Click Here

sylantechelecri
Not applicable

Sylantech Electricals are the leading LED manufacturers in India that focus on the consumer's needs and saving the environment. We provide all varieties of led lights for various purposes like Indoor commercials, Outdoor commercials, Office use, Home use, Fancy LED, Smart LED. Now we are here with new Rechargeable LED Bulbs

To know more click here

faheemcs2019
Not applicable
Greetings I am so glad I found your website, I really found you by error, while I was researching on Google for something else, Regardless I am here now and would just like to say thank you for a incredible post and a all round enjoyable blog (I also love the theme/design), I don’t have time to read through it all at the moment but I have book-marked it and also added in your RSS feeds, so when I have time I will be back to read much more, Please do keep up the awesome work. http://www.adverts.pk/-1/posts/1_Vehicle_Sale_Purchase/0/
Baburaja
Not applicable

Very nice blog. Thanks for sharing. You may also like Schools In India & Best Schools In Delhi

Deepakkumar
Not applicable

Wow sir Apne hame bhut hi Achi jankariya Di jai sir.

 

Bajrangisoch.com

 

Kya ap Instagram par follower bdhana cute hai to hamre link par click kare.

 

Instagram par follower kaise bdhaye.click here.

lesacote
Not applicable

Thanks for the awesome and amazing content. Administrative law assignment help is happy to help you. 

KROXINE
Not applicable
KROXINE
Not applicable

per rendere la loro giornata più allegra! Preparati a far sorridere i tuoi amici con queste fantastiche battute.
Queste sono solo alcune delle frasi da dedicare, ma ti ricordo che nel menù in alto ne puoi trovare tante altre in base al tipo

Bank IBPS PO IBPS Clerk IBPS SO IBPS RRB PO IBPS RRB Clerk SBI PO SBI Clerk SBI SO RBI GRADE B BANK RBI ASSISTANT NABARD SSC SSC CGL SSC CHSL SSC CPO SSC JE SSC MTS SSC Stenograher SSC GD INSURANCE ESIC SSO ESIC UDC GIC Assistant Manager LIC AAO LIC ADO NICL OICL Teaching TET CTET KVS NVS Railway Railway ALP Railway Group D RPF Exams RPSF RRB ASM RRB JE RRB NTPC STATE LEVEL Uttar Pradesh UP Police Constable UP Police SI UPPCL UPSSSC LMRC VDO Gram Panchayat Adhikari Lekhpal Delhi Delhi Police DMRC DSSSB Madhya Pradesh MP Patwari Rajasthan RSMSSB Rajasthan Police Haryana Harayana Police HSSC Group D Bihar Bihar Police Constable BSPHCL Punjab PSPCL Government ID Aadhar Card PAN Card Voter ID Passport Driving License Fast Tag Cast Certificate Birth Certificate Updates PRL Recruitment 2019 at PRL RES in Physical Research Laboratory Jobs RBI Office Assistant Pre Admit Card 2020 SBI Clerk PET Admit Card 2020 Mahadiscom Vidyut Sahayak Recruitment 2020 37 Vidyut Sahayak Vacancy Indian Union Budget 2020 Admin Top Current Affairs for Sarkari Naukri Preparation 01 February 2020 Karnataka Bank PO Scale 1 2020 Admit Card Chhattisgarh PSC 2020 Admit Card High Court AP Recruitment 2020 at HC AP NIC in on 1 February 2020 Rajasthan HC Stenographer 2020 Online Form
KROXINE
Not applicable
vinay
Not applicable
I love neutral topic like Instagram.
JoshRussel
Not applicable

Thanks for posting this article. I've read about this earlier here https://themesnap.com/how-to-choose-a-logo-designer/

jasonwalter
Not applicable

 

Spoiler
 

Our selection of commercial playground equipment, site amenities, and shade structures can fit any budget, theme, or size restriction. Get a quote today!
jumping pillow | 

 

anna91
Not applicable

This is a very interesting post and all the details are in this post are awesome.

hotmail login

anna91
Not applicable

Things are very open and intensely clear explanation of issues. was truly information. Your website is very beneficial. gmail login

faheemcs2019
Not applicable

Complete info. I'm new reader , I just reach there by chance. I found this blog useful and genuine

http://www.translationservicesworld.com/legal-documents.html

madyjones
Not applicable

Nice blog! thank you for the shared. 

best wet dog food
best dog food for small dogs
saaifalee
Not applicable

if you would like to improve your knowledge only keep visiting
this site and be updated with the latest news posted here.

http://www.mobile-phone.pk/