Diamond Price Calculator

To answer our question, we used the Gemstone Price Dataset from Kaggle which originated from ggplot2. The dataset contains 53940 rows and 11 columns: Unnamed 0, carat, cut, color, clarity, depth, table, price, x, y, and z. We cleaned the data. Any rows with dimension values of 0 or duplicates were removed for being irrelevant, leaving us with 53755 rows. Finally, all columns except for carat, cut, color, clarity, and price were removed because we only needed these columns to answer our research question. Each of the 4 C’s have a specific value. For the features that were categorical variables, we found an online 1-8 scale to convert the values into numbers. We did this for the Cut, Clarity, and Color variables.

The dataset ended with 53755 rows and 8 columns. In order to get a better understanding of our cleaned dataset, we explored it with some visualizations. First, we graphed the scatter plot of carat vs price. For the color, cut, and clarity graphs, we used a box plot to prevent the categorical data from clumping around a few numbers on the x-axis. Interestingly, when the color quality increased, the price actually decreased. Next we made a correlation matrix. Carat and price had a value of 0.92, which symbolized a very high correlation.

We assessed three different models to fit the data. Our first model was a multilinear regression model using the 4 C’s as features. We performed linear regressions between price, carat, cut, color, and clarity. After observing that the graph had an upwards curve, we used a second-degree polynomial linear regression model since we believed it would be a better fit. Using data split into an 80% train set and 20% test set, we fit the linear regression model on it and outputted our equation:

Price = -5707.877 + 3773.190(carat²)+ 145.228(cut_num) + 358.693(color_num) + 613.223(clarity_num)

Train R² Score: 0.841

Therefore, we chose a random forest model. The random forest model is better because it is able to capture non-linearities better than a standard linear regression model. For example, increasing the carat from 0.5 to 1.0 will not affect price the same way as an increase from 2.5 to 3.0, but on the linear regression model, these are treated as the same interval. It also requires little tuning and resists overfitting because each tree sees a slightly different dataset; they make different errors and averaging their results reduces variance. The random forest algorithm can also learn combinations of features. For example, large carat diamonds with high clarity get exponentially more expensive. This model works by bagging and training many different trees on random subsamples of the training data (drawn with replacement). Then, the model averages each tree’s predictions. Using 500 estimators, and a max depth of 10, we arrived at a

Test R² = 0.98

These results were much better than the linear regression model as the R² score was larger.