If you really want to build a product that scales to 100 million users, it will be a real big gun and I am sure you will face many challenges both from a technical and non-technical point of view.
Technical challenges:
- Advanced machine learning algorithms
- Scaling
Non-technical challenges:
- Acquiring customers
- Keeping the product in line with what customers need
These challenges may sound trivial, but as a tech guy, I can say with confidence that scaling to 100 million people is definitely not a piece of cake. When you have a hundred thousand users, you are all good. When a hundred million users are using your app, things start breaking and you really need some groundbreaking technology to handle this volume of traffic.
However, if you are looking for something simple - a Quora like question and answer website, then that’s really simple and I have done it (with the help of a couple of friends). With some effort, anyone can do it.
Here are the screenshots:
The technology behind it:
- Django (Python) based backend
- Frontend in Bootstrap
- MySQL as database
- Hosted on Heroku (using my free tier of Heroku)
- Used Google OAuth for authentication
- Tinymce for forms and formatting
- Rake for basic NLP/keyword extraction
The database schema is described below:
- UserProfile table:
- user = models.OneToOneField(User, primary_key=True)
- avatar = models.ImageField(null=True, upload_to=generate_filename, default="../default_avatar.png")
- bio = models.CharField(max_length=50, null=True)
- followers = models.ManyToManyField(User, related_name='following')
- following = models.ManyToManyField(User, related_name='followers')
- Topic table:
- name = models.CharField(max_length=50)
- url = models.CharField(max_length=100, primary_key=True)
- followers = models.ManyToManyField(User, related_name='topic_followers')
- Questions table:
- text = models.CharField(max_length=100)
- time = models.DateTimeField(default=timezone.now)
- asked_by = models.ForeignKey(UserProfile, on_delete=models.SET_NULL,
- null=True, db_index=True)
- url = models.CharField(max_length=100, primary_key=True)
- details = models.CharField(max_length=200)
- topics = models.ManyToManyField(Topic, related_name='topic_questions')
- followers = models.ManyToManyField(User, related_name='question_followers')
- Answers table (yo is the term I used for upvote :P):
- question_url = models.ForeignKey(Question, on_delete=models.CASCADE)
- answered_by = models.ForeignKey(
- UserProfile, db_index=True, on_delete=models.SET_NULL, null=True)
- question_text = models.CharField(max_length=100)
- text = models.TextField()
- time = models.DateTimeField(default=timezone.now)
- yoers = models.ManyToManyField(User, related_name='yoers')
- Comments table (answers have comments):
- text = models.CharField(max_length=200)
- time = models.DateTimeField(default=timezone.now)
- commented_by = models.ForeignKey(
- User, on_delete=models.SET_NULL, null=True)
- answer = models.ForeignKey(Answer, on_delete=models.CASCADE, db_index=True, related_name='comments')
An interesting piece remains - newsfeed algorithm:
As can be seen, the newsfeed has 3 parts:
- Latest Questions/Answers:
- user = request.user
- latest_questions = Question.objects.all().order_by('-time')[:FEED_COUNT]
- latest_answers = Answer.objects.all().order_by(
- '-time')[:FEED_COUNT].prefetch_related('question_url')
- latest_qa = list(latest_questions) + list(latest_answers)
- latest_qa.sort(key=lambda x: x.time, reverse=True)
- yo_list, yo_count_list = utils_get_yo_info(latest_qa, user)
- latest_qa_with_yos = zip(latest_qa, yo_list, yo_count_list)
- return render(request, 'home/latestqa.html', {
- 'latest_qa_with_yos': latest_qa_with_yos,
- 'user': user,
- 'domain': settings.DOMAIN_NAME})
Here is a brief explanation of the code above:
- Generate a list of latest 20 or so questions
- Generate a list of latest 20 or so answers
- Sort them in the reverse order of time after combining the 2 lists
- Show them as the latest Q/A newsfeed
- Topics you like:
- topics = request.user.topic_followers.all()
- topic_questions = list(set(Question.objects.filter(
- topics__in=topics).order_by('-time')[:FEED_COUNT]))
- topic_questions.sort(key=lambda x: x.time, reverse=True)
- yo_list, yo_count_list = utils_get_yo_info(topic_questions, request.user)
- topic_questions_with_yos = zip(topic_questions, yo_list, yo_count_list)
- return render(request, 'home/topicsyoulike.html',
- {'topic_questions_with_yos': topic_questions_with_yos})
This is quite straightforward - a simple database query + some manipulations
- People you follow:
- following = request.user.following.all()
- answers = list(set(Answer.objects.filter(
- answered_by__in=following).order_by('-time')[:FEED_COUNT]))
- answers.sort(key=lambda x: x.time, reverse=True)
- yo_list, yo_count_list = utils_get_yo_info(answers, request.user)
- answers_with_yos = zip(answers, yo_list, yo_count_list)
- return render(request, 'home/peopleyoufollow.html',
- {'answers_with_yos': answers_with_yos})
Again, this is also quite straightforward to understand.
All of these newsfeed ‘algorithms’ were written by me and are in a very raw form. None of them is really ‘intelligent’ and I am sure many of the database queries can be optimized significantly. But again, this wasn’t supposed to be Quora, it was rather supposed to be Quora-like.
Note that the site has been abandoned. It was just an academic project (no commercialization) and the questions and answers you see in the screenshots are because it was tested by my friends a couple of days back. It took a few weeks to complete the web-app.
The name SoclWebApp was randomly chosen (
Socl == (Soc)ialize + (L)earn). The site isn’t indexed by search engines and you won’t be able to find it on Google/elsewhere. However, I would be more than happy to answer any questions :)